Macan EV with Thermal Vision custom project

[Macan EV turbo driver]

I once owned a 2014 Rolls Royce Wraith with thermal vision, and spotting kangaroos off the side of the road, totally outside the high beam vision (which was also excellent in the RR Wraith) was easy. There was always sufficient time to brake quickly, turn off high beam (which dazzles kangaroos, making them panic) and safely pass with the kangaroo safely staying still.

The RR Wraith only had deer as an equivalent warning for kangaroos. However, being well accustomed to kangaroos, their image on infrared, was easy to spot.

I have been waiting to be able to buy this much needed safety option, ever since selling my RR Wraith.

Sponsored

 

[BostonSteve]

Ha - no there's 0 hackery going on here. In fact 0 car-side software changes. This is basically just a fancy dash cam that connects to my phone. I'm using an existing pathway to allow phones to project to the car's display surfaces.

What are you thinking? If I had control over the headlight module I'd probably light up detected collision risks.

Can you point me at something that shows how to do this?

 

[isahouh]

I wonder why Porsche doesn't allow the infrared "Night Vision" camera setup in the Macan (ICE or EV), when you can get it in every other 4-door Porsche. And why doesn't Porsche have dual sun visors for both front seats of either Macan while you can get it in the Cayenne.

I would've added it as an option to my build if it was available!

But alas... but then again now I have something I have full control of and can customize however I'd like powered by state of the art hardware + ML models

I suspect changing their stack to Android Automotive OS probably had something to do with it. Harder to re-use prior night vision hardware/software.

 

[Heavywun]

I wonder why Porsche doesn't allow the infrared "Night Vision" camera setup in the Macan (ICE or EV), when you can get it in every other 4-door Porsche. And why doesn't Porsche have dual sun visors for both front seats of either Macan while you can get it in the Cayenne.

to get you to buy a Cayenne of course

 

[sor]

Can you point me at something that shows how to do this?

This is the most interesting part (or at least the most obscure part if you wanted to DIY this) and maybe there’s a reason why it isn’t covered as in depth as the rest. Only a guess but I wonder if it is isn’t something similar to how CarPlay and Android Auto send map graphics to this area of the driver display. I thought this was somewhat controlled by regulation and App Store or developer entitlements (don’t want anyone to just write an app that lets you watch YouTube while driving). If this is the method it’s almost certainly Android auto because Apple won’t even let you dev without preapproval.

 

[alvaro]

I’m not clear on how this was integrated to the PCM

Edit: ah —- android

 

[W1NGE]

I once owned a 2014 Rolls Royce Wraith with thermal vision, and spotting kangaroos off the side of the road, totally outside the high beam vision (which was also excellent in the RR Wraith) was easy. There was always sufficient time to brake quickly, turn off high beam (which dazzles kangaroos, making them panic) and safely pass with the kangaroo safely staying still.

The RR Wraith only had deer as an equivalent warning for kangaroos. However, being well accustomed to kangaroos, their image on infrared, was easy to spot.

I have been waiting to be able to buy this much needed safety option, ever since selling my RR Wraith.

Suspect it's the same as was available on the BMW given parentage of same name?

 

[Petzi]

the question is: how to „project“ anything to the dashboard?

 

[sor]

Perhaps the wrong place to discuss the technical aspects but I was wondering why you were limiting yourself to 13ms, versus something like batching several frames and processing those or choosing every N frames. Regardless it seems like you’re getting good results with YOLO, but I wonder if it is necessary to reclassify objects at frame rate speeds, or if a different model may do better if you had a higher budget, or if you could reclaim some compute power.

 

[Macan EV turbo driver]

With the Rolls, the thermal imaging projected to the centre screen. If a kangaroo did jump onto the road in front of the car, an image of a deer projected onto the HUD.

 

[isahouh]

Perhaps the wrong place to discuss the technical aspects but I was wondering why you were limiting yourself to 13ms, versus something like batching several frames and processing those or choosing every N frames. Regardless it seems like you’re getting good results with YOLO, but I wonder if it is necessary to reclassify objects at frame rate speeds, or if a different model may do better if you had a higher budget, or if you could reclaim some compute power.

Found the engineer The blog post reflects an earlier version of the code. The architecture is now fully decoupled with independent pipelines.

• Camera thread (60 fps) —> writes to ring buffer
• Ring buffer —> inference thread (variable FPS - in practice 60 fps) —> mailbox
• Ring buffer + mailbox —> streaming thread (still playing with this… currently streaming at 40 fps)

So if inference takes too long for whatever reason it doesn’t block the pipeline. The 13ms thing was from an earlier synchronous version where everything had to complete within the time budget for a single frame.

In practice inference happens fast enough that this is rarely actually needed but there are occasional spikes now and then that this handles (I profiled p50, p95 times, etc…)

I’m not batching because I want this processing as close to real-time as possible and inference time isn't a bottleneck. At 65 mph 100 ms batch collection time = 9.5 feet travelled. If there’s a deer on the road I want to know ASAP. Each track maintains an exponentially decayed confidence; hits add, misses decay. Crossing a confirm threshold promotes to “real” (green box in the demo video); dropping below maintain prunes it (still need to tweak this).

These Orin Nanos are surprisingly efficient. I'm about to add kalman filtering to the object tracking though... will see if that starts straining things...

 

[jwatte]

You're running billions of FMAs on the matrix multiplies, and the low-dimensional Kalman filter is a concern?
(Although, to be fair, the MMs use the GPU, whereas that Kalman probably uses the CPU, but those ARM cores are still pretty beefy, all told.)

But, you know, any excuse to upgrade to a Thor

 

[isahouh]

So technically its not the Kalman part that’s most computationally taxing it’s the o(n3) Hungarian matching I have on top (and now we’re firmly in mumbo jumbo tech speak land )

I don’t really expect this to be an issue unless if for some reason there are dozens of tracks.

 

[Petzi]

Ha - no there's 0 hackery going on here. In fact 0 car-side software changes. This is basically just a fancy dash cam.

What are you thinking? If I had control over the headlight module I'd probably light up detected collision risks.

idid read your blog (respect!).
but you have not explained how you get this on the dash? with all respect but this seems to be a big challenge. def. nobody had managed something like this jet.
„this is basically a fancy dashcam“ ?? nobody here is able to install a dashcam app in the pcm. (not even porsche for their own dashcam) not to mention the center display in the dash..

 

[sor]

idid read your blog (respect!).
but you have not explained how you get this on the dash? with all respect but this seems to be a big challenge. def. nobody had managed something like this jet.
„this is basically a fancy dashcam“ ?? nobody here is able to install a dashcam app in the pcm. (not even porsche for their own dashcam) not to mention the center display in the dash..

It looks like we got what we will get on this. Existing smartphone integration APIs can do this, and there are examples of apps (dozens?) putting content on the driver UI. It isn't that nobody has the tools to do something like this, but if a dash cam company tried to distribute an app that plays video in front of the driver it would be insta-denied. Not the intended use of these APIs and safety regulation involved. It is even possible that if this gets enough attention those APIs might get locked down more. An actual instructional might get more personal attention.

So I understand why there's not a lot of detail on this point, and it will likely remain a private dev app, but at the same time it is too cool not to share.

Sponsored