What is a NeRF?: Teaching AI to “Imagine” a 3D World Like a Ghost

Introduction: The Regret of 2D Photos

Imagine you took a picture of a delicious cake with your smartphone. The photo looks perfect, but it is ultimately just a flat surface. If you want to see if there are strawberries on the back of the cake, or check the thickness of the cream from the side, staring at the photo won’t help because the information from those angles was never recorded.

In the past, if you wanted to view an object from different angles, you needed expensive 3D scanners, or you had to pay a designer to painstakingly “sculpt” a model on a computer. But now, a rising star has emerged in the field of Artificial Intelligence—NeRF (Neural Radiance Fields).

Simply put, a NeRF is like a painter with an extraordinary imagination. You only need to show it a few photos of a cake taken from different angles, and it can construct the entire 3D shape of the cake in its “mind.” Afterwards, you can observe the cake from any angle you wish, or even generate a hyper-realistic VR video.

Core Concept: How Does NeRF Work?

To understand NeRF, let’s use an analogy from daily life: “Ray Casting” vs. “The Ghost Painter.”

1. Traditional 3D Modeling: Like Making a Sculpture

Traditional 3D modeling (like characters in video games) is akin to molding clay figures. The computer needs to know how many triangles make up the object’s surface, and textures are pasted onto it like stickers. This requires a lot of geometric data and struggles with things that don’t have clear “surfaces,” like smoke, transparent glass, or complex fur.

2. The NeRF Approach: Like Capturing Light

NeRF doesn’t sculpt; it thinks more like a physicist studying light. It asks: “If I stand at this specific spot, what color of light will hit my eye (or camera)?”

NeRF imagines the entire space as being filled with countless tiny particles (think of the air as being filled with an extremely thin mist).

• Location: Every point in space has coordinates (X, Y, Z).
• Direction: From which angle are you looking at this point?
• Density: Is this point “solid” or “empty”? (e.g., the cake is solid, the air is empty).
• Color: What color does this point emit or reflect?

3. The AI’s “Imagination” Process (Training)

This is like teaching a ghost painter to draw.

1. Input: You show the NeRF 20 photos taken around the cake and tell it exactly where each photo was taken.
2. Guess & Correct: The NeRF closes its eyes and tries to “render” (draw) an image as seen from that same angle. At first, it draws a messy blur.
3. Compare: It compares its drawing with the original photo you provided and realizes: “Oops, I painted this part too red, and that part should be empty space.”
4. Learn: It uses a Neural Network to adjust its internal parameters.
5. Repeat: This process is repeated thousands of times until it can perfectly reproduce all the photos you gave it.

At this point, it hasn’t just memorized the photos; it has effectively learned the laws of light and object distribution throughout that entire space.

NeRF’s Superpowers: Why Is It Revolutionary?

NeRF represents a massive breakthrough compared to traditional technologies:

• Handling “Translucent” and “Reflective” Objects:
  Traditional 3D models struggle to depict the refraction behind a glass of water or the hazy look of smoke. NeRF is naturally good at this because it calculates the journey of light rays through space. If a light ray passes through a sparse area (like smoke), it accumulates a little color; if it hits a solid area, it blocks the view. This makes NeRF-generated images achieve Photorealism.

• No Expensive Equipment Needed:
  You don’t need a LIDAR scanner worth tens of thousands of dollars. Theoretically, by taking a video while circling an object with your smartphone, you can generate a high-quality 3D scene.

• Tiny Storage Space:
  Although NeRF represents complex 3D worlds, it doesn’t need to store huge geometric files. All the information is compressed inside the “brain” of a neural network (often just a few MB in size).

Recent Progress: From “Turtle Speed” to “Lightning Fast”

When NeRF was born in 2020, it had one huge flaw: Speed. It was slow. Training a single scene could take a graphics card a whole day, and rendering one image could take several minutes. This made it unusable for real-time games or mobile apps.

However, the pace of technology is astonishing. Recent advancements (such as Instant-NGP or 3D Gaussian Splatting) have completely changed the game:

1. Speed Boost: Modern techniques can finish training in seconds or milliseconds.
2. Real-Time Rendering: You can now explore NeRF-generated scenes in a web browser at a smooth 60 frames per second, just like playing a high-end video game.
3. Dynamic Scenes: Early NeRFs could only handle static objects. Current tech (like Dynamic NeRF) can even reconstruct dancing people or flowing water.

Future Applications: How Will NeRF Change Our Lives?

1. Immersive Maps (Google Immersive View):
   If you’ve used the latest Google Maps, you might have seen the feature that allows you to fly around landmarks freely. This is an application of NeRF technology. It turns flat street-view photos into 3D city models, making you feel like you are actually there.

2. E-commerce and Virtual Try-Ons:
   In the future, buying shoes won’t mean looking at a few stiff pictures. You will be able to rotate the shoe freely, see the texture of the sole clearly, and even use AR to see how it looks on your own foot.

3. VR and the Metaverse:
   If we want to transport the real world into a virtual one, NeRF is currently the simplest and most realistic path. Want to turn your living room into a VR chatroom? Film a video, upload it, and NeRF will handle the rest.

4. Movie Visual Effects:
   Hollywood can produce realistic 3D scenes at a lower cost, reducing the reliance on green screens and laborious manual modeling.

Conclusion

NeRF represents a turning point in computer vision: shifting from “geometry-centric” to “light-centric.” It teaches AI how to understand the play of light and shadow just like the human eye. Although it is still evolving, the next time you see a lifelike 3D virtual showroom on your phone, remember that behind the scenes, a NeRF might be weaving light rays in real-time just for you.

How AI Understands the World of “Points”? — Introducing PointNet

Imagine you have a very intelligent AI robotic dog. You show it a picture of a cat, and it instantly recognizes it as a “cat.” You play a recording of a dog barking, and it immediately identifies it as a “dog.”

However, if you hand it a 3D printed model of a mug, or let it wear LiDAR glasses to look at the real 3D world, it might suddenly become “dumbfounded.”

Why? Because the data format of the 3D world is vastly different from photographs (2D pixels). And one of the key technologies enabling AI to truly understand the 3D world is the protagonist of our story today — PointNet.

Part 1: What is a “Point Cloud”?

To understand PointNet, we first need to know what it processes. It deals with something called a Point Cloud.

Don’t be intimidated by the name. Imagine you are holding an invisible apple in your hand. You want to describe the shape of this apple to someone, but you have no pen and no clay. You only have a magic wand that can create tiny glowing dots in the air.

You tap thousands of dots on the surface of the apple. When you remove the invisible apple, those thousands of glowing dots remaining in the air roughly outline the shape of the apple.

This is a Point Cloud. It is simply a pile of scattered points in 3D space (each point has X, Y, and Z coordinates) that collectively represent the shape of an object. The LiDAR used by self-driving cars scans the surrounding environment and produces countless points just like this.

Part 2: The Challenge AI Faces — The Trouble with Disorder

For AI, understanding these points poses a huge challenge, which we call Unorderedness.

Example:
Suppose you want to teach an AI to recognize a “Hamburger.”

• If it’s an image (pixels): The top left of the image must be the bun, the middle is the meat, and the bottom is the lettuce. The order is fixed. If you shuffle the pixels, the image turns into static noise, and the hamburger is unrecognizable. Traditional AI (like Convolutional Neural Networks) is very good at handling these fixed grids.
• If it’s a point cloud: Recall the “air apple” earlier. You can tap the top of the apple first, then the bottom; or the left side first, then the right. No matter where you tap first, as long as you don’t move their positions, these points still form an apple together.

For a computer:

• Point Cloud A: Input Point 1, Point 2, Point 3…
• Point Cloud B: Input Point 3, Point 1, Point 2… (Only the input order has changed)

Although Point Cloud A and Point Cloud B represent the exact same apple, traditional AI algorithms would think these are two completely different sets of data! It’s like telling an AI that “Tomato Scrambled Eggs” and “Eggs Scrambled with Tomato” are two different dishes—the AI would crash on the spot.

PointNet was born to solve this exact “ordering” problem.

Part 3: The Magic of PointNet — Symmetric Functions

Proposed in 2017, PointNet was the first deep learning network to directly process these unordered points. How did it achieve this?

It uses a very clever but simple mathematical principle: Symmetric Functions.

Sounds sophisticated? You actually use it every day.

Life Analogy: Throwing Coins into a Piggy Bank

Imagine a group of elementary school students (representing the points), each holding coins of different values (representing the information of the points). Now we want to calculate how much money the class has donated in total.

1. The Traditional Clumsy Method: Make the students line up by student ID number and record “Student #1 donated $5, Student #2 donated $1…”. If the line gets shuffled, the record sheet becomes a mess.
2. PointNet’s Method (Symmetric Operation): The teacher brings a giant piggy bank.
   • The students don’t need to line up.
   • Tom can throw his money in first, then Jerry; or Jerry first, then Tom.
   • The result is the same! The total amount in the piggy bank never changes.

In, PointNet, this “piggy bank” operation is called Max Pooling.

The workflow of PointNet is roughly like this:

1. Solo Operation (MLP): First, look at each point individually and enrich the information of this point (e.g., not just knowing where it is, but inferring whether it likely belongs to the handle or the bottom of the mug).
2. Piggy Bank Operation (Max Pooling): No matter how many points you have or what order they come in, I only extract the most significant feature from all points (similar to the logic of the total sum, but here it takes the maximum value).
3. Final Recognition: Through these extracted “strongest features,” the AI can judge: “Oh! No matter how you shuffle the order, this pile of points definitely looks like a mug!”

Part 4: No Need to Tilt Your Head — The Alignment Network (T-Net)

PointNet has another trick up its sleeve.

If you turn a mug upside down or tilt it, many “dumber” AIs won’t recognize it as a mug anymore.

PointNet designed a little assistant called T-Net.
This is like the AI having a pair of auto-correcting glasses. When it sees a crooked point cloud of a mug, T-Net automatically calculates a rotation angle to “straighten” the mug in its “mind.”

In this way, no matter how the object is rotated or placed, PointNet can first rotate it to a standard pose before identifying it.

Conclusion: Why is PointNet So Important?

Before PointNet, scientists had to take detours to process 3D data:

• Either flatten 3D into 2D photos (losing depth information).
• Or chop the 3D space into countless tiny cubic grids (voxels), like in Minecraft, which consumes huge amounts of computing power, making the computer slow as a snail.

PointNet is the first warrior to “face the chaos directly.” It doesn’t need to flatten data, nor does it need to chop grids. It swallows raw, unordered point cloud data directly, and it is extremely fast and effective.

Today, when you see:

• Self-driving cars precisely avoiding pedestrians;
• FaceID scanning your facial structure;
• Robot vacuums building a map of your home;

Behind them, there is often PointNet or its evolved versions (like PointNet++) silently contributing wisdom. It has given machines the true ability to understand the three-dimensional world.

What is PCRNet? The Magical “Jigsaw Puzzle” Skill for Robots

Imagine holding two torn pieces of a treasure map in your hands. You want to put them together to reveal the complete map. What does your brain do? You rotate one piece, compare the jagged edges, and adjust them until they fit perfectly.

In the fields of Artificial Intelligence (AI) and Robotics, there is a technology that does something very similar. It is called PCRNet.

1. The Basic Concept: What is a “Point Cloud”?

To understand PCRNet, we first need to know what it works with—the Point Cloud.

• Real-life Analogy:
  Imagine you are in a pitch-black room, quickly waving a laser pointer over a cup. You can’t see the cup clearly, but every time the laser hits the surface, a dot of light appears. If you wave the laser fast enough and densely enough, these thousands of dots come together to roughly outline the shape of the cup.

  This is a “Point Cloud.” For self-driving cars or robot vacuums, their “eyes” (LiDAR sensors) don’t see colorful photographs like we do. Instead, they see a 3D world made up of thousands of tiny floating dots.

2. The Core Problem: The Puzzle Headache

This leads to a difficult problem: Point Cloud Registration.

• Real-life Analogy:
  Let’s suppose you are a robot vacuum.

  • Second 1: You see a sofa in front of you (this is the “Source Point Cloud”).
  • Second 2: You move forward a step and turn slightly. Now, you see the sofa from a different angle (this is the “Target Point Cloud”).

  In the robot’s brain, these are two different 3D images. Even though it’s the same sofa, its position and angle don’t match. Point Cloud Registration is the process of perfectly overlaying these two images to tell the robot: “Hey, this is the same sofa as before, you just moved.”

  If registration fails, the robot might think there are suddenly two sofas in the room, or it might crash into a wall while trying to find its charging dock.

3. What is PCRNet?

PCRNet (Point Cloud Registration Network) is a model that uses Deep Learning (AI) to automatically solve this “jigsaw puzzle.”

Before PCRNet, older methods (like the famous ICP algorithm) acted like an extremely stubborn perfectionist. They tried to align the images by nudging them bit by bit. If the two images were too far apart to begin with, these methods often failed or were very slow.

PCRNet, on the other hand, is like an experienced puzzle master. It doesn’t just blindly nudge pieces around. Because it has studied millions of puzzles during its training, it can look at the shape of a piece and immediately judge: “Aha! This piece needs to be rotated 30 degrees and placed right here.”

How PCRNet Works (Simplified)

We can break PCRNet down into two main steps:

1. Feature Extraction: aka “Finding Key Points”

   • It doesn’t stare at every single boring dot. Instead, it looks for unique shapes, like the armrest of the sofa or the corner of a table. Even if you rotate the object, these “features” are still recognizable.
   • Analogy: It’s like looking for a friend in a crowd. You don’t deduce who they are by looking at their pores; you look for key features like “wearing a red hat” or “blue jacket.”

2. Pose Prediction: aka “Guessing the Move”

   • After extracting features from both views, PCRNet compares them. Its neural network “brain” then calculates and tells you directly: “Move the first image $X$ meters to the left and rotate it $Y$ degrees, and it will match the second image perfectly.”

4. Why is PCRNet Powerful?

The “Net” in PCRNet stands for Neural Network. Its uniqueness lies in:

• Speed: Since it’s based on Deep Learning, once it’s trained, it calculates the result extremely fast.
• Robustness: It can handle images that are initially far apart (unlike traditional algorithms that get confused easily).
• Smart Iteration: There is an upgraded version called Iterative PCRNet (i-PCRNet).
  • Analogy: Standard PCRNet takes one look and snaps the pieces together. i-PCRNet snaps them together, looks closely and thinks, “Hmm, it’s slightly off,” and then fine-tunes it until the fit is seamless.

Summary

Simply put, PCRNet is an AI technology that gives machines “Spatial Imagination.”

It helps self-driving cars know exactly where they are on the road, it helps robotic arms precisely pick up parts from a conveyor belt, and it even helps doctors overlay 3D CT scans taken at different times to compare a patient’s progress.

It takes two chaotic, misaligned views of a 3D world and, through intelligent calculation, turns them into a coherent, unified whole.

This is English content.
Welcome to my blog. This is a multi-language test post.

Splitting Strategies for Indexing Code Files in RAG and Choosing Vector Databases

Retrieval-Augmented Generation (RAG) systems need to split code into suitable “chunks” when processing code files to generate vector embeddings and retrieve them efficiently. The characteristics of code files (such as syntax structure, function/class boundaries, comments, etc.) determine that splitting strategies cannot simply apply the rules of text documents. Below, I will detail splitting strategies, vector database recommendations, and specialized methods for code based on current best practices (latest guidelines in 2025). Information comes from professional articles and community discussions.

1. Recommended Document Splitting Strategies (Chunking Strategies)

The goal of code splitting is to preserve semantic integrity (e.g., not breaking up function definitions) while controlling chunk size (usually 512-2048 tokens to avoid embedding model overload). General strategies are suitable for beginners, but for code, optimization is needed to handle indentation, syntax, etc.

General Splitting Strategies (Applicable to code, but need adjustment):
Implement using frameworks like LangChain or LlamaIndex. These strategies can maintain context continuity through overlap (10-20%).

Optimization Tips:

• Chunk Size and Overlap: Recommended for code blocks: 512-1024 tokens, overlap 100-200 tokens to capture cross-block dependencies.
• Add Metadata: Attach file path, line number, language type to each chunk for filtering retrieval.
• Handling Structured Content: Code often has comments/strings; handle separately to avoid noise; test multiple strategies and evaluate retrieval accuracy (e.g., using RAGAS framework).

2. Recommended Vector Database Types

Vector databases are used to store embeddings of code chunks (e.g., generated using CodeBERT or text-embedding-3-large models). Selection criteria: support for ANN (Approximate Nearest Neighbor) search, high throughput, metadata filtering. All mainstream DBs are suitable for RAG, but prefer open-source/easy-to-integrate ones.

Recommended Types and Examples:

Selection Guide:

• Small Scale: Chroma (runs locally, no server needed).
• Large Scale: Milvus (supports distribution, suitable for code search).
• Indexing Strategy: Use HNSW or IVF-PQ indexing, dimension matching embedding model (e.g., 768 dimensions). Evaluation metric: recall > 0.8.

3. Specialized Splitting Strategies and Vector Databases for Code Files

Yes, there are specialized methods! Code is not plain text and requires consideration of AST (Abstract Syntax Tree) to preserve structure. General DBs can also be used, but specialized tools are more efficient.

Specialized Splitting Strategies:

• AST-based Chunking: Use Tree-sitter or ANTLR to parse code and generate AST, then split by node (function, class, module). Retain complete function body + comments, avoid cross-function splitting. Example: LangChain’s CodeSplitter or custom script, splitting by def/class boundaries.
• Graph-based Chunking: Treat code as a graph (nodes=functions, edges=calls), chunk as subgraphs. Use HyDE (Hypothetical Document Embeddings) to expand queries during retrieval.
• Tool Recommendation: LlamaIndex’s CodeSplitter (supports Python/JS, etc.), or Sourcegraph’s Cody (built-in RAG for code).

Specialized Vector Databases/Extensions:

• There is no purely “code-dedicated” vector DB, but general DB + code plugins are excellent:
  • Weaviate: Built-in code module, supports AST embedding and structured search.
  • Zep or ZincSearch: Combined with Elasticsearch vector plugin, optimized for code semantic search (supports fuzzy matching variable names).
  • Specialized Systems: Sourcegraph (vector + full-text search for code repos), or GitHub Copilot’s backend (based on custom vector index).
• Why specialized is better: Code retrieval needs to handle symbol similarity (e.g., variable renaming), specialized methods can improve recall by 20-30%.

Implementation Suggestions

1. Start: Use LangChain + Chroma + Recursive Chunking to test small codebases.
2. Evaluate: Use RAGAS to measure faithfulness and relevance.
3. Resources: Refer to GitHub’s RAG_Techniques repository to experiment with multiple strategies.