Core ML is a machine learning framework introduced by Apple back in 2017. It allows you to integrate pre-trained machine learning models into your iOS app projects, so your users can enjoy features like image recognition and natural language processing. It’s also able to maximize your app performance while reducing power consumption.
Keep reading for our guide to integrating machine learning with Core ML into your iOS app and the advantages and challenges of using it.
What is Core ML, and what is it used for?
Core ML is a cutting-edge machine learning framework that lets you integrate pre-trained machine learning models into iOS, macOS, watchOS, and tvOS applications. It enables developers to add intelligent features to their apps without needing deep expertise in machine learning. The main goal is to keep your app both efficient and responsive.
Core ML lets you use advanced hardware in your apps, like CPU, GPU, and Neural Engine. That way, you can get the best performance possible from your app while keeping your power consumption low. Models are run directly on a user’s device to keep data private while making the app both fast and reliable.
You can create and train models with the Create ML app, which is included in Xcode. With it, you can build custom models that are formatted for Core ML and ready to be integrated in your app.
So, what can you do with Core ML?
The framework allows you to perform tasks that would be too time-consuming for you to code manually. For example, when you integrate Core ML into your app, it can:
- Categorize photos - Can recognize different types of images so users can organize their photo libraries. (Ex. Apple’s Photo app already does this).
- Detect objects - Uses image recognition to identify different objects in photos (Ex. shopping apps can identify products from user-uploaded images and provide instant information).
- Process natural language - Analyzes text input and offers smart responses or even personalized recommendations (Ex. keyboards can suggest the next word based on context).
- Real time object tracking - When integrated with ARKit, it allows you to continuously detect and track objects in the Augmented Reality environment using Core ML, providing real-time annotations or interactions based on the detected objects.
Advantages and challenges of Core ML
Below are the main advantages and drawbacks of implementing a Core ML framework.
The advantages
- Data privacy
Processing happens locally on the user’s device, meaning data remains private. Because data isn’t sent to external servers, users can rest easy knowing their private information is safe.
- Offline availability
Since data is not sent to the cloud server, it allows it to function locally without an internet connection, which is beneficials to various scenarios like cataloging vacation photos in airplane mode or identifying plants in remote areas.
- Low Latency and Real-Time Results
Because Core ML uses CPU, GPU and Neural Engine, you can maximize app performance by eliminating the need for network requests. This is particularly useful for applications that require immediate processing, such as video analysis or image recognition.
- Seamless integration
Building training models is easy when using Create ML and Xcode. Core ML also supports pre-written machine learning models, which you can import and convert for your app.
- Personalized to user preferences
Because you can carry out on-device model retraining and fine-tuning, your app can adapt depending on the user’s preferences, allowing you to provide much more tailored experiences.
The challenges
- Slower loading times
If your machine learning model is too large, your app may face slower loading times, and it will eat away at a user’s memory usage. This is mainly an issue for older devices that don’t have the CPU to deal with it.
- Battery consumption
Running complex machine learning tasks on a mobile device can be computationally intensive, leading to increased battery usage. This may be particularly noticeable on older devices
- Complex model training
While you can integrate pre-built machine learning models into your app, the process can still be complicated. As a developer, you need to have a decent understanding of machine learning principles to create models that work.
- Platform limitation
Core ML is restricted to Apple devices, which limits its use in cross-platform development scenarios
- Model updates
Keeping the on-device model up-to-date with new user data can be challenging. Even after releasing an app update with a new model, not all users will immediately update their apps. This can lead to fragmentation, with different users running different versions of your mode. Also updating the app with a new model may strain the user’s network connection, especially if the model is large
How to integrate a Core ML model into your app
Below is a comprehensive guide on integrating the MobileNetV2 Core ML model into a real-time ARKit app for image classification. This tutorial will combine elements from the Apple examples and best practices for AR and Core ML integration.
Set Up Your Project
- Create a new Xcode project with the "Augmented Reality App" template.
- Select SwiftUI for the interface and ARKit for the content technology.
- Download the MobileNetV2 model from Apple's Machine Learning Models page.
Add the Core ML Model
- Drag the downloaded MobileNetV2.mlmodel file into your Xcode project.
- Ensure "Copy items if needed" is checked and your target is selected.
Set Up Core ML and ARKit
- Create a new Swift file called ARClassificationView.swift and add the following code:
swift
import SwiftUI
import ARKit
import Vision
struct ARClassificationView: UIViewRepresentable {
func makeUIView(context: Context) -> ARSCNView {
let arView = ARSCNView(frame: .zero)
arView.delegate = context.coordinator
arView.session.delegate = context.coordinator
let config = ARWorldTrackingConfiguration()
arView.session.run(config)
context.coordinator.arView = arView
return arView
}
func updateUIView(_ uiView: ARSCNView, context: Context) {}
func makeCoordinator() -> Coordinator {
Coordinator(self)
}
class Coordinator: NSObject, ARSCNViewDelegate, ARSessionDelegate {
var parent: ARClassificationView
var classificationRequest: VNCoreMLRequest?
var arView: ARSCNView?
init(_ parent: ARClassificationView) {
self.parent = parent
super.init()
setupVision()
}
func setupVision() {
guard let model = try? VNCoreMLModel(for: MobileNetV2(configuration: MLModelConfiguration()).model) else {
fatalError("Failed to load Core ML model")
}
classificationRequest = VNCoreMLRequest(model: model, completionHandler: handleClassification)
}
func session(_ session: ARSession, didUpdate frame: ARFrame) {
guard let classificationRequest = classificationRequest else { return }
let pixelBuffer = frame.capturedImage
let orientation = CGImagePropertyOrientation(rawValue: UInt32(UIDevice.current.orientation.rawValue))!
let handler = VNImageRequestHandler(ciImage: CIImage(cvPixelBuffer: pixelBuffer), orientation: orientation)
DispatchQueue.global(qos: .userInitiated).async {
try? handler.perform([classificationRequest])
}
}
func handleClassification(request: VNRequest, error: Error?) {
guard let results = request.results as? [VNClassificationObservation] else { return }
if let firstResult = results.first {
DispatchQueue.main.async {
self.showClassificationResult(firstResult.identifier, confidence: firstResult.confidence)
}
}
}
func showClassificationResult(_ identifier: String, confidence: Float) {
// Here you would update your AR scene with the classification result
print("Classified as: \(identifier) (Confidence: \(confidence))")
}
}
}
Update ContentView
- Replace the contents of ContentView.swift with:
swift
import SwiftUI
struct ContentView : View {
var body: some View {
ARClassificationView()
.edgesIgnoringSafeArea(.all)
}
}
Visualize Results in AR
- To display classification results in AR, add the following method to the Coordinatorclass:
swift
func showClassificationResult(_ identifier: String, confidence: Float) {
DispatchQueue.main.async {
// Remove previous result nodes
self.arView?.scene.rootNode.enumerateChildNodes { (node, _) in
if node.name == "resultNode" {
node.removeFromParentNode()
}
}
// Create a new text node
let textGeometry = SCNText(string: "\(identifier)\nConfidence: \(String(format: "%.2f", confidence))", extrusionDepth: 1.0)
textGeometry.firstMaterial?.diffuse.contents = UIColor.white
let textNode = SCNNode(geometry: textGeometry)
textNode.name = "resultNode"
// Scale and position the text node
textNode.scale = SCNVector3(0.002, 0.002, 0.002)
textNode.position = SCNVector3(0, 0.1, -0.5)
// Add the text node to the scene
self.arView?.scene.rootNode.addChildNode(textNode)
}
}
Optimize Performance
- To improve performance, you can reduce classification frequency:
Add a property to track the last classification time and only classify every 0.5 seconds:
swift
var lastClassificationTime = Date()
func session(_ session: ARSession, didUpdate frame: ARFrame) {
guard let classificationRequest = classificationRequest else { return }
let currentTime = Date()
guard currentTime.timeIntervalSince(lastClassificationTime) > 0.5 else { return }
lastClassificationTime = currentTime
// ... rest of the method
}
Results
This guide demonstrates how to integrate the MobileNetV2 Core ML model into a real-time ARKit app for image classification. The app continuously classifies what the camera sees and displays the results in augmented reality. Remember to test the app on a physical device, as the simulator doesn't support ARKit.
Below is a sample of output from our testing where it correctly classified the desk as seen in the image below.
Final thoughts
If you’re a newbie to machine learning models, integrating a Core ML model into your iOS app project can seem nerve-wracking. However, following these straightforward steps will help you speed up your coding process so you can provide your users with a smooth and personalized experience.
If you need help at any stage in the app development process, contact Tapptitude today. We’re app-building experts with years of experience helping developers create and publish their dream applications.
FAQs
Can I use Core ML with SwiftUI?
You can integrate CoreML with SwiftUI by creating a SwiftUI view that interacts with your CoreML model and serves as the user interface. This view lets you see input fields, buttons, and visual components that display predictions or results.
What type of machine learning models are compatible with Core ML?
Core ML supports plenty of different model types. You can integrate classification, regression, and object detection models, amongst others. You can import models that are made using popular frameworks, including TensorFlow, PyTorch, and scikit-learn, but only if you convert them to the Core ML format.
Is there any documentation available for Core ML?
Apple has a number of comprehensive documents for Core ML, including guides and sample projects, as well as API references. You can find all the resources that you need to create machine-learning iOS projects on the Apple Developer website.
How can I test whether my Core ML model is accurate?
The best way to test whether your CoreML model is accurate is by using a separate validation dataset to compare your model’s predictions with its actual outcomes. Carrying out this testing will help you assess how well the model is performing and identify areas where you can optimize your code.
Where can I get help with integrating Core ML into your iOS app?
If the Apple Developer website isn’t providing enough information, consider reaching out to your developer experts at Tapptitude. We specialize in app building and will be able to optimize your iOS project with CoreML and machine learning capabilities.
Tapptitude
Tapptitude is a mobile app development company specialized in providing high-quality mobile app development services, and a top-rated app company on Clutch.
