Issue Counting Score

I'm currently creating a sort of Pong clone as practice, but I've run into some troubles.

I have some code separated into two different scripts in which when the ball hits a racket, the racket will gain one point. However, if the ball hits a colored wall, the racket on the same side of the wall looses a point.

However, I've noticed something very odd: Whenever I keep the rackets still, the scoring system works as expected: https://i.gyazo.com/7c9d9756f2a3bfe9f7a720296d382684.mp4

But if I begin moving a racket, and the ball is below said racket, that racket will gain a seemingly random amount of points without even touching the ball: https://i.gyazo.com/515a54bb35f24e614a797d3d2d90252a.mp4

I've checked the colliders in real-time on both the ball and the rackets, and nothing seems out of the ordinary. I just don't understand what's causing this issue.

Here's the script attached to the ball GameObject:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using UnityEngine.UI;
5.  
6. public class Ball : MonoBehaviour {
7.     public float speed = 30;
8.     public Text timeText;
9.     public AudioSource gainSound;
10.     public AudioSource lossSound;
11.     public AudioSource other;
12.  
13.     // Use this for initialization
14.     void Start ()
15.     {
16.         // Initial Velocity
17.         GetComponent<Rigidbody2D>().velocity = Vector2.right * speed;
18.     }
19.  
20.     float hitFactor(Vector2 ballPos, Vector2 racketPos,
21.                     float racketHeight)
22.     {
23.         return (ballPos.y - racketPos.y) / racketHeight;
24.     }
25.  
26.     void Update()
27.     {
28.         if (timeText.text == "0")
29.             {
30.                 timeText.text = "Game Over";
31.             }
32.     }
33.      
34.  
35.     void OnCollisionEnter2D(Collision2D col)
36.     {
37.         if (col.gameObject.name == "RacketLeft")
38.         {
39.             gainSound.Play();
40.             // Calculate hit Factor
41.             float y = hitFactor(transform.position,
42.                                 col.transform.position,
43.                                 col.collider.bounds.size.y);
44.  
45.             // Calculate direction, make Length=1 via .normalized
46.             Vector2 dir = new Vector2(1, y).normalized;
47.  
48.             //Set Velocity with dir * speed
49.             GetComponent<Rigidbody2D>().velocity = dir * speed;
50.  
51.         }
52.  
53.         // Hit the right Racket?
54.         if (col.gameObject.name == "RacketRight")
55.         {
56.             gainSound.Play();
57.             // Calculate hit Factor
58.             float y = hitFactor(transform.position,
59.                                 col.transform.position,
60.                                 col.collider.bounds.size.y);
61.  
62.             // Calculate direction, make Length=1 via .normalized
63.             Vector2 dir = new Vector2(-1, y).normalized;
64.  
65.             // Set Velocity with dir * speed
66.             GetComponent<Rigidbody2D>().velocity = dir * speed;
67.         }
68.  
69.         if (col.gameObject.tag == "Loss")
70.         {
71.             lossSound.Play();
72.         }
73.  
74.         if (col.gameObject.tag == "Other")
75.         {
76.             other.Play();
77.         }
78.     }
79. }
80.  

Here's the script attached to the rackets:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using UnityEngine.UI;
5.  
6. public class MoveRacket : MonoBehaviour {
7.     public float speed = 30;
8.     public string axis = "Vertical";
9.     public Text score;
10.     private float scoreAmount = 0;
11.  
12.     void FixedUpdate()
13.     {
14.         float v = Input.GetAxisRaw(axis);
15.         GetComponent<Rigidbody2D>().velocity = new Vector3(0, v) * speed;
16.     }
17.  
18.     private void OnCollisionEnter2D(Collision2D collision)
19.     {
20.         scoreAmount = scoreAmount + 1;
21.         score.text = scoreAmount.ToString();
22.     }
23. }

I understand that this problem may or may not be too involved (I'm a beginner so I'm not sure what would qualify as complex), but any help at all would be greatly appreciated!

Edit: Forgot to include that this is a two player game, with one player controlling the racket with the arrow keys and another player controlling the other racket with WASD.

 

dude, you are hitting the top and bottom white blocks, that's your scoring since your "OnCollisionEnter2D" from the rackets doesn't check if it's the ball the one hitting

 

Oh, duh. Now I feel stupid...
Thanks for the help.

 

Also it seems to make more sense to add the score Detection on the ball itself. Tag each wall with Player1Score and Player2Score. (Player1Score Wall should be behind Player2's Paddles) then if you collide with these objects you play your lossSound.Play and update the scores in the ball script