Recently, I embarked on a personal experiment to test how some of the current AI models could assist in automatically detecting a car’s make, model, etc based only on some images. This experiment aimed to evaluate how well different AI models (both locally deployable and enterprise solutions) can identify car details and match data fields in our system. It wasn’t an high priority project, but rather an exploratory venture.
The Concept: AI-Driven Field Selection via Images
The goal was straightforward: use an AI model to analyse car photos and automatically detect attributes such as make, model, and color using the References from our Database. While existing systems rely on a car’s national code or it’s VIN for identification, accessing or remembering the nat code can be inconvenient. I wanted to see if an AI could handle the task using only a photo.
Challenge: Gathering Enough Images
A key obstacle was sourcing enough car images for testing. GDPR regulations ruled out using existing customer data, so explicit consent was required for any images. I began by asking friends and family for car images. Most were fine with it, though some asked that identifying details like license plates be blacked out, which I did.
For more variety, I approached a nearby car dealer who graciously allowed me to photograph 45 of their cars, bringing the total number of images in this collection to 62. This effort had its own bonuses; each car had a sheet displayed on it that confirmed make, model and model specifications.
Each car was photographed from a side-to-front angle. No motorcycles were included, though the dataset did include some utility vehicles and trucks. Some identifying marks were blackedout to ensure the it was genuinely analyzing the images.
Technical Approach: Testing the AI
A car image is processed and sent to a chosen AI model with a structured prompt.
- The AI first identifies the car’s make from a list of available make types.
- Once the make is identified, the script retrieves the models for that make, allowing the AI to pinpoint the matching car model.
- and this step is also subsequently happening for the specific model specifications.
- As a proof of concept, it also attempts to auto-fill additional details about the car, such as its color.
The technical workflow involves fetching required data from a database, such as make names, models, model specifications, and possible car colors. The car photo is read, encoded into Base64 format, and sent to the AI model within a JSON payload. Depending on the chosen system (OpenAI or LocalAI), the image along with the structured prompt is sent via an API request. The AI’s response is then cleaned and parsed to extract relevant details.
If the AI is unable to accurately identify the color, make, model, or sub-model details of the car, it defaults to an “others” category. The identified details are then saved in JSON format and aggregated into a comprehensive results file.
OpenAI GPT-4o (Enterprise)
For this experiment, the GPT-4 multi-modal AI is being used, with an API key obtained from OpenAI’s website. Although a newer model called GPT-o1 offers even more advanced features, it is currently difficult to access because it requires Tier 5 API access.
Results
The performance of GPT-4o in car identification tasks is impressive. It achieved perfect make type identification, high accuracy in model identification (56 out of 62), and identified secondary model specifications with reasonable success (30 out of 62) despite complexity. Further it correctly identified the color for 59 out of 62 cars, with some errors likely caused by lighting conditions in the photos.
Model specifications were marked as correct only if it was the best fit for the given context. This means that I either chose no model when there were no fitting options, or selected a specific model when it was unequivocally the best match. I marked a specification as incorrect if it was entirely wrong or if there were clearly better alternatives available.
This process cost around 1.45 cents per request, totaling 90 cents for all 62 photos, based on OpenAI’s token pricing at the time of writing. However, this cost could have been significantly reduced if the prompts had been written more efficiently without unnecessary token usage.
LLaVA 1.6 with Mistral 7B (Deployed Locally)
For running a local instance, LLaVA 1.6 combined with Mistral 7B, which became available on September 27, 2023, was utilized. Although Mistral’s multimodal Pixtral 12B, a newer version expected to offer better performance, has been introduced, I currently lack the hardware capabilities to test it.
Setting up LLaVA 1.6 with Mistral 7B is straightforward. First, LM Studio must be installed. Then, the model should be downloaded and loaded. Afterward, the server can be directly initiated within the platform.
Results
The Results here are disappointing. It correctly identified only 3 makes. It successfully matched just 2 models. For model specifications, it managed to correctly identify just 1. Although, it did identify the color for roughly 25 of the cars.
All 62 images took several hours to process due to hardware constraints and was less accurate, despite the benefit of zero costs for processing.
Despite the efforts to fine-tune and improve it, the results were not promising. Its ability to identify a Mercedes from an image without any additional data was adequate but not impressive. Nonetheless, this outcome was notably better than the subpar results achieved when using the motornetzwerk fields.
Future Improvements
To improve the experiment, several adjustments could be made:
- Use of better Locally Deployable AI Models: Using a more powerful Locally runable Model with better hardware could have returned better results for the Local Processed Prompt. For instance, utilizing the newly released Multimodal Ai Pixtral 12b from Mistral.
- More Photos Per Car: Including multiple perspectives for each car could improve the accuracy. More images would maybe allow the it to make more informed decisions.
- Expand Fields: Incorporating additional data fields beyond the initial four categories, such as engine type and door count.
- Optimise Prompts: Refining the prompts and query structure to make the processes more efficient and cost-effective.
- Larger Sample Size: A larger sample size could offer a better evaluation. While 62 samples provided some insight, a bigger dataset may have offered a better result.
Although it was just a test, this experiment provided useful insights into how AI could be applied to practical tasks like auto-filling car data.
Experimenting with AI-Models for Auto-Filling Car Data was originally published in willhaben Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.