Scene lighting is messed up in AR foundation using LWRP

Hi, I am trying to augment a western scene which has various assets and complex animations through an iOS device onto the real world. I am using AR foundation (version 2.1.0) with Light Weight Render Pipeline(version 5.6.1) to build the scene in iOS device (ARKit version 2.1.0). I used the SampleLWRPscene example code provided in this link:
https://github.com/Unity-Technologies/arfoundation-samples/tree/lwrp-support-2019.1-with-2.1, and just replaced the game object in placed prefab on the tracked plane to the scene prefab that I built.

I built the app and the scene appears at the touch location, but the rendering gets messed up whenever I try to scale the AR session origin or the prefab, it gives weird colors. Please find attached the image below. When I used a directional light in the hierarchy and another directional light in the scene prefab without scaling anything, it shows up fine but when I change the scale or disable any light, the scene on AR messes up. I am not understanding what exactly is happening. I need help in understanding this issue and coming up with a solution.

All I wanted to do is try to change the scale of the scene so that it appears to the required scale in the real world without effecting the lighting on the scene.

Thanks in advance