Oracle OEM Agent 12C High Usage of CPU on Solaris

That Oracle OEM Agent 12C caused spiking and high usage of CPU on Solaris is a very common issue. I did experienced it couple of times. To solve it, oracle support document “12c EM Agent CPU spiking and High Usage on Solaris with many Database and Listener Targets (Doc ID 1536871.1)” almost offers the complete solution. I just include it here for quick reference only.


12c EM Agent CPU spiking and High Usage on Solaris with many Database and Listener Targets (Doc ID 1536871.1)


In this Document

Symptoms

Cause

Solution

References


APPLIES TO:

Enterprise Manager for Oracle Database – Version 12.1.0.2.0 and later
Oracle Solaris on SPARC (32-bit)
Oracle Solaris on SPARC (64-bit)

SYMPTOMS

12c EM agents are observed to consume high CPU and create high spikes on Solaris boxes when many Database and Listener targets are managed with the same EM agent. The CPU consumption and spike are directly proportional with the number of CPU that the physical server has (the more CPUs are there, the higher is the agent CPU usage).

If you would like to discuss the issue raised in this note with others in the EM Community, please reply to the “My Oracle Support Communities” thread captured below:

CAUSE

This was investigated under the following BUG:
Bug 15953286 – agent high cpu usage when scheduling metrics for many targets at once

The CPU consumption is generated by many factors as:

– a known issue described in Note 1427773.1 – The Cloud Control Agent 12c Status Timeout / Connection refused / “security.pkcs11.P11SecureRandom.implNextBytes”
– The EM agent scheduler does not spread the Response metric or other metrics within the schedule interval and it’s trying to start all of them in the same time (creating the spikes)
– EM agent uses UCP (Universal Connection Pool) for managing the database connections which is suffering from BUG 10203435 where about 24 threads are started for each created pool (with the BUG patch it will create only 4)
– a JDBC leak BUG 13583799
– too many garbage collector threads for the agent’s JVM defined by default.
– a default behavior of Solaris Hotspot JVM’s implementation where a more expensive thread controlling mechanism (LWP) is being used instead of direct Kernel Calls.

SOLUTION

1. Ensure the default values are set in agent_inst/sysman/config/emd.properties as:
     agentJavaDefines=-Xmx128M -XX:MaxPermSize=96M
2. Implement the Java security fix as per note 1427773.1
3. Agent DEBUG level to be set to WARN (higher tracing levels will add an overhead on CPU usage).
4. Include the following lines in agent_inst/sysman/config/s_jvm_options.opt file.

-XX:ParallelGCThreads=4
-XX:-UseLWPSynchronization

(or apply patch 16398691 on the EM agent, which is implementing these changes.)
5. Apply JDBC leak patch 17591700 on the EM agent ORACLE_HOME.

6.  sym-link-ing the libmtmalloc.so.1 library (as root): ln -s /usr/lib/sparcv9/libmtmalloc.so.1 /usr/lib/secure/sparcv9/libmtmalloc.so.1
7. Run: export LD_PRELOAD_64=libmtmalloc.so.1
8. Install patch 10203435 for the UCP thread leak BUG on the Agent’s ORACLE_HOME
9 Install the MLR patch 16303155 that contains the fixes for bug 16072609 and bug 16298267 which are spreading the metric schedules on wider intervals
10. Modify the following property in agent_inst/sysman/config/emd.properties file:
from
_ResponseDelay=30
to
_ResponseDelay=60
11. If does not exist, add the following parameter: _scheduleSpecSeed=300 as well in emd.properties

12. Restart the EM agent after all the above operations.

REFERENCES

BUG:15953286 – AGENT HIGH CPU USAGE WHEN SCHEDULING METRICS FOR MANY TARGETS AT ONCE

Posted in My Reference | Tagged | Leave a comment

Initialization Parameter “Compatible” in oracle database

With oracle database upgrade from 11.1.0.7 to 11.2.0.3, we experienced an issue while we created materialized view within. The error is

image

It looked like oracle 11gR2 introduced new feature of materialized view, such as fast refresh, which requires oracle initialization parameter “compatible” to be set to 11.2.x.x.x instead of 11.1.0.0.0.

And then we conducted this change on database and rebooted it with setting of this parameter to 11.2.0.3.  That change made the creation of materialized view  successful. If the change is missed on standby database, if you have, you will get error ORA-16778: redo transport error for one or more database primary.

Besides, the following backup setting also need to be done to have the RMAN backup working properly. Otherwise, RMAN backup failed with related errors.

image

Posted in My Reference, Oracle Case Study, Oracle Point | Tagged | Leave a comment

How does oracle sequence AUDSES$ work?

In oracle database, sessions connecting to an Oracle database have a number of identifiers assigned to them which can be found in the V$SESSION dynamic view. One of these identifiers is V$SESSION.AUDSID. This identifier is particularly useful because it can also be obtained by a session using the SQL function USERENV(’SESSIONID’).  In the way, a session can find its own session-rated information. Oracle support Doc ID 122230.1 discloses “how a session gets its V$SESSION.AUDSID value”.

The value that is assigned to AUDSID is taken from a sequence in the Oracle Data Dictionary, the sequence SYS.AUDSES$. When AUDSID is assigned to a new session the SYS.AUDSES$ sequence is incremented using NEXTVAL and the new value is assigned to the session.

However, not all sessions get an AUDSID: Background sessions such as SMON increment the SYS.AUDSES$ sequence and the value of their AUDSID is 0.

Oracle 10.1 onwards, SYS sessions are further distinctive: 
If AUDSID=0, then it is an internally generated SYS session. 
If AUDSID=UB4MAX(4294967295), then it is a direct SYS session.

If we do further study, I found the most confused part of sequence AUDSES$ is setting of CACHE_SIZE. 

The value increase of AUDSES$ is generally defined by “INCREMENT_BY”, which is 1 for this sequence. But, it also depends on “CACHE_SIZE”. The default setting for “CACHE_SIZE” was 20, and was too low for a RAC or high login rate system. Lower setting of “CACHE_SIZE” can cause slowness and even hangs. Therefore, at oracle 10.2.0.3 onwards, that’s changed to 10000.

If “CACHE_SIZE” is set to 10000, then 10000 sequence values after value of “LAST_NUMBER” will be stored in memory for faster access and avoid possible enq:SQ-contention. However, “LAST_NUMBER” jumps to 10000 higher value before using up cached 10000 values, and new 10000 sequence values are then loaded to memory. That’s why the “LAST_NUMBER” is increased by 10000 and we are seeing the gap of using sequence number.

The setting of CACHE_SIZE to ‘NOCACHE’ means the none of sequence values are stored in memory, which force the increment of this sequence by 1 strictly. However, this option may sacrifice some performance.

Posted in My Reference, Oracle Point | Tagged | Leave a comment

The Comic Pals: The Cardiovascular Adventure

 Chapter 1

The Preparation

“Okay, you two know why we’re here, right?” Mark inquired as he paced around the lab. Bob and Joe nodded vigorously.

“Good. Just as an extra heads up, we’ll be exploring the respiratory and cardiovascular systems up close. In fact, we’ll be actually in someone’s body.”

“I forgot,” Joe interrupted. “Whose body are we going into?”

“Steve’s.”

“Does Steve even know?”

“No.”

“How are we going to get in?” Bob asked.

“Excellent question, Bob. I’m glad you asked. Using my newly invented shrink ray, I’ll shrink us down to a microscopic size, and then we’ll enter through Steve’s mouth. Did anybody bring anything?”

“I brought my climbing gear,” Joe answered, hefting his grappling hook.”

Mark nodded approvingly. “Excellent move. We should all expect difficult terrain in the body, what with all the split paths and valves.”

“I brought SNACKS!!!!!!!!!!” Bob squealed, waving a jumbo bag of Lays™ Original flavor potato chips in the air.

Mark and Joe stared at Bob.

Somewhere in the distance, a cricket chirped.

Finally, Mark broke the silence. “Well, I guess that would be useful, since we’ll need to eat. Anything else?”

Bob and Joe kept quiet. “Okay then, I also brought some stuff.”

Mark rummaged in a nearby box and removed three things that looked suspiciously like pistols, three potions, and three masks that covered everything except the eyes. He held up the goggles. “These are magical potions, specially designed by me. When drunk, the drinker will teleport back to the lab and grow him back. Use this as a worst-case scenario backup.”

“What are the pistols for?” Joe wondered.

“Those are laser guns, also specially designed by me to fire two different types of lasers, explosive or electric,” Mark answered. “The final item is the Gas Mask 4000, which provides unlimited oxygen while worn.”

“Ooooooooh,” Bob and Joe oohed.

“Now, is everybody ready?” Mark asked.

“Yes.”

“Okay,” Mark confirmed. “Advance to the Shrinkray Seats!”

Bob and Joe sat down in the chairs in front of the shrink ray. Mark pressed some buttons on his control panel and joined the duo. The shrink ray sparked with high voltage, then fired out a blinding green ray that enveloped the three kids. The lab flashed, and an explosion sounded. When the light died down, the trio stood up and examined themselves.

“Wow, I feel like Jack!” Joe exclaimed. “Add a giant and this will feel like a fairy tale!”

Just then, Mom walked past the lab to do the laundry.

“Feel like a fairy tale now?” Bob teased.

“Okay, let’s go,” Mark said, distributing the items. Joe examined the laser guns with a look of interest on his face.

“Okay, gang, to the living room!”

 

A Couple of Minutes Later

 

Minutes later, the trio was standing on the chin of Steve. Mark turned around. “Let’s review the safety instructions. Follow my orders if necessary. If anything attacks, shoot it with the laser gun. If you see one of us drop unexpectedly or you think you need it, put on the Gas Mask 4000 immediately. And most importantly, stay with the group.” Mark looked up. “Bob? Anyone seen Bob?”

“WOOHOOOOOOO!!!!!!!!!!” Bob’s voice sounded from Steve’s mouth. Mark and Joe turned their heads just in time to see Bob’s head disappear behind Steve’s lower incisors.

“BOB!!!!!!!” Mark screamed. “WEREN’T YOU LISTENING?!?!?!?!?!?” He ran to Steve’s mouth and soon had also gone into his throat.

Joe looked at the scene, then ran forward, his hand gripping tightly to his laser gun………

The Mouth

The mouth is one of the main parts of the respiratory system. You need it to breathe in oxygen and to exhale the carbon dioxide that your body doesn’t need. This body part also allows you to talk, to chew and to swallow. The mouth is mainly used in the digestive system, but I’m supposed to focus on breathing, so yeah.

When you breathe, your neck, chest, and abdomen muscles work together to expand your chest cavity, drawing oxygen into your lungs. The mouth doesn’t warm oxygen like the nose does, so it’s better to breathe through the nose, but in a pinch the mouth will do.

There are three “tubes” connected to your mouth called salivary glands that produce saliva, or “spit”, as some people call it. That’s not too important, but good to know anyway.

The mouth is truly important to us and every bit is vital for our survival.


 

Chapter 2

A Sticky Situation

“What were you thinking?!” Mark scolded.

“That sliding down teeth is fun!” Bob grinned, showing no sign of remorse or guilt.

The group was walking on Steve’s tongue, their voices echoing off the saliva-covered walls of his mouth.

“You have to stay with the group! We need to be ready for anything!”

“Fine.”

Mark walked past the sublingual gland. “We should not only be watching for danger ahead, but also from above! We—“ He paused. His foot was submerged in a brownish green substance.

“Eeeeeeeeeeeeeeeew,  snot!” Joe cringed.

Mark examined the substance. “I do believe that this may be mucus.” He tried to dislodge his foot. His foot refused. “My foot is stuck.”

“Here, I’ll help!” Bob offered, running forward. He tripped over a papillae and fell headfirst into the mucus.

“Oh, great,” Mark groaned. “Joe? Now would be a good time to use the laser gun. You should set it to explosive.”

Joe aimed his weapon. “Do I just pull the trigger?”

“Yes.”

Joe pulled the trigger. A red beam of light shot out of the front. As soon as the laser made contact with the mucus, it detonated, blasting Bob and Mark into the epiglottis.

“Oh, come on!” Joe protested. “How come I always have to keep chasing after you guys?” He ran after Mark and Bob, making sure to avoid the mucus that was still dripping down.


 

The Nose

What’s that snarky thing beneath your eyes? What do little kids pick and eat? What contains the ability to smell?

The answer, of course, is the nose, and here we’re going to look at it in full detail.

The nose contains two types of cells: goblet and epithelial. The epithelial cells have a substance called cilia that kind of looks like tiny, white hairs on their surfaces. The goblet cells make mucus, the sticky substance that appears in your nose from time to time.

When you inhale through your nose, the oxygen passes through a chamber called the nasal cavity, where it is warmed to a good temperature. Meanwhile, the mucus will trap the non-oxygenic particles while the oxygen remains untouched. The cilia will then transport the mucus into the upper airways, where you can now do anything with it.

The nose functions incredibility well, and is often overlooked.


 

Chapter 3

Down The Hatch!

“Well, that was rather effective,” Mark noted, examining the point of impact where the laser had landed. The mucus was nowhere to be found. “I may have miscalculated on the explosion radius.”

“At least the mucus is gone,” Bob pointed out.

“True. We should keep going.”

“Left or right?” Bob wondered, looking at the two paths ahead.

“Which one is closer to the epiglottis?”

“The hippowhat?”

“The flap.”

“Left.”

“Then left,” Mark confirmed.

“Why can’t we go right?” Joe asked.

“The right is the larynx, which leads to the stomach. We’re trying to enter the trachea.”

“Oh.”

The group slowly walked into the trachea, unaware that from the larynx, something was watching them. It chuckled, split into two, and floated after the trio.

The Trachea

As the oxygen travels through the nasal cavity and past the salivary glands, it enters a pipe that leads to the lungs. This pipe is known as……… the trachea.

The trachea, or windpipe, has a cartilage flap at the top called the epiglottis. This flap remains open for oxygen until you swallow, in which case it will close. This is why your parents tell you not to talk while chewing. You’ll end up breathing as you swallow, and so your epiglottis will stay open, causing your swallowed food, or bolus, to get into the lungs. Then your lungs will send a message to your brain telling you to choke, and you’ll choke.

The trachea isn’t more than just a passageway from the nose and mouth to the lungs, but it still serves an important purpose.


 

Chapter 4

A Fork In The Road

The group trudged on, determined to reach their destination. “How long is this trapeza thing?” Bob asked.

“It’s TRACHEA, not trapeza, and it should end soon.”

“Can we stop and rest for a while?” Joe asked.

“Sure. I bet we’re all feeling beat.” Mark agreed.

“Why not eat a bit as well?” Bob suggested.

The motion passed unanimously.

Bob opened a plastic bag of Kellogg’s™ corn flakes.

 

?                    ?                    ?

“Well, that was good for all of us,” Mark said, standing up. “We should try to cover some more ground before night falls.”

The group continued their journey.

“Do you get the feeling that we’re being watched?” Bob asked uneasily.

“And do you get the feeling that we’re being watched by something green?” Mark turned, spotting a green splotch just before it vanished

“Hey, look! The path splits here!” Joe stopped and looked back. “What are you looking at?”

Mark dismissed the thought. “Nothing. Why don’t we go right?”

The group entered the bronchial tubes, looking more grim than ever………


 

The Lungs and Bronchial Tubes

After a while, the trachea will split into two tubes. Guess what they’re called? Duh. I’m not even going to answer this one.

The bronchial tubes carry the oxygen deep into the lungs, splitting as they go. Eventually they end, but by now they have already filled up the spongy containers. At the end of each bronchi, as they are scientifically termed, there is an air sac called the alveoli. The oxygen groups in these sacs, ready to be distributed into the blood. The lungs contain spots for blood to flow in, oxygenate, and go, leaving the carbon dioxide behind to be exhaled.

The ribs prevent direct contact with the lungs from the outside world, so lung problems usually happen from the inside. Even if you tried, right now, you would only feel your heartbeat.

The lungs and bronchial tubes are the most important parts of respiration, and without them, you probably would not last long.


 

Chapter 5

The rift

“Which way are we supposed to go?” Bob asked. “The path keeps splitting and splitting.”

“All of the paths lead to the same place,” Mark answered. “They all end at an air sac.

“Are you talking about this air sac right here?” Joe joked, blowing a bubble.

“Of course not.” A voice boomed out from behind them. The group slowly turned around.

Slithering toward them was a large army of green amoeba creatures. The one in the front spoke again. “We are the Contaminators! You will all face infection!”

“Infection! Infection!” The creatures chanted.

“Everybody, guns out! Start shooting!” Mark yelled, whipping out his weapon. Joe and Bob drew their weapons as well and started firing. The group shot straight and accurately, but for each Contaminator down,  two took its place.

“There’s too many of them!” Joe called.

“We have to get into the alveoli!” Mark screamed over the pandemonium. “Masks on!”

The group sprinted to an air sac, their hearts pounding out of their chests. Bob and Joe jumped in immediately, but Mark turned and fired one more bolt before joining them. The trio closed their eyes as they transferred into the blood.

Blood

The red liquid that flows out of cuts and wounds is more than just red liquid. Some of it is made of particles that are definitely not red………

The average adult has 5 liters of blood, and it consists of three kinds of cells. There are red blood cells, white blood cells, and platelets. There are millions upon millions of these cells in your blood, all floating in the main substance, plasma.

Red blood cells, or erythrocytes, are shaped like pancakes, which gives them more space. Each one contains hemoglobin, a sort of protein that sticks to oxygen and gives the cells their red color.

White blood cells, or leukocytes, mainly exist to patrol the circulation and fight infection and diseases. Some of them will break down tired red blood cells and send the useful products to the bone marrow to reproduce.

Platelets will just go along with the circulation until a blood vessel is damaged. They will then rush to that vessel and group together, causing a chemical reaction of fibrin strands crossing, which will trap more platelets and the previously said erythrocytes. This will create a clot, and over time the vessel will heal. Platelets don’t have a nucleus and so will only last about 8 to 12 daays.

Blood is needed to deliver oxygen to your cells. Along with bringing oxygen, blood also picks up waste chemicals and sends them to the kidneys to be removed.

Blood is but the pedestrian of the blood vessels, but should not be overlooked.


 

Chapter 6

The Alpha

“That was close.” Joe remarked.

“Well, we gotta keep going,” Mark replied. “Strange, I haven’t spotted any cells. They should be plentiful in here.”

“Coming through!” A red blood cell flowed by, knocking Bob over.

“Artery C3 is damaged!” A tissue-like platelet sped past.

“A germ was spotted in vein H7!” A white blood cell disappeared into the blue tunnel where the blood seemed to be flowing.

“Never mind.”

A red blood cell stopped. “Hi. Are you lost? You don’t seem like cells.”

“We’re travelers from outside this body,” Mark explained.

“You mean there’s more of us?! That’s a lot of new friends to make. What are you doing here, anyway?”

“We wanted to explore the circulation up close.”

“Well, it seems nobody told you about the traffic in here,” the cell noted. “Want me to give you a ride?”

“Are you sure about this?” Mark checked. “We don’t want to disrupt your business.”

“Of course!” the erythrocyte replied. “Helping people makes me feel warm and good.”

The group climbed onto their ride. “What’s your names?” he asked.

“I’m Mark, this is Bob, and over there is Joe.”

“Do you have a name?” Bob asked the cell.

“Of course. My name is Red.”

“Are all of the red blood cells named that?” Joe wondered.

“No, I’m just given that name by the Omega Marrow.” Red flowed into the blue tunnel they had seen the other cells go into.

“Interesting. The cells are alive,” Mark noted. “This appears to be the pulmonary vein.”

“Good observation,” Red approved.

“What’s that thumping up ahead?” Bob asked.

“The Alpha, or what you know as the heart. She controls us and sends us to the correct areas.”

Just as Red finished answering Bob’s question, the tunnel suddenly opened up into a large chamber. A female voice echoed through the atrium. “Group R9, please take path C48.”

“This the best part,” Red whispered to the group.

The valve in the ground spun open, and all of the cells dropped in. Bob whooped as Red spun in dizzying circles into another room underneath. The valve above closed, and the room expanded for takeoff.

The Heart

Sitting right there in the middle of the rib cage, pumping all 5 liters of blood around and around, over and over, is the master of the body. The heart is the leader of the cardiovascular system, and it has control over all others.

The heart is actually two pumps, and each one has an atrium and ventricle. Scientists (and everyone else, for that matter) call them the left atrium, the left ventricle, etc. etc.

The right atrium receives the deoxygenated blood from the inferior vena cava and superior vena cava, and sends it to the right ventricle, which pumps it to the oxygen-packed lungs, where it gets oxygenated and sent to the left atrium. This chamber will deliver the now oxygenated blood to the left ventricle, which will pump the blood through the aorta and through the circulation again.

When you listen to your heartbeat, you are actually hearing the valves between your atriums and ventricles opening and closing. The bum-bum that you hear is actually one beat, not two, so if you’re worried about your heartbeat being too fast, you may have counted wrong.

The heart is the anchor of the body, and should it fail, we are all doomed.


 

Chapter 7

Delta

“What’s it like to be a blood cell?” Mark asked Red.

“Just normal. Constantly going around in circles, passing on the oxygen, over and over.”

“Doesn’t it ever get boring?”

“Nope. There’s always something interesting going on.”

“Hey there, Red!” A white blood cell floated over. “Who’s your companions?”

“This is Bob, Joe, and Mark. Guys, meet Delta.”

“Nice to meet you.” Delta bobbed with the plasma current. “Are you visitors? You don’t seem like cells.”

“Yeah, we’re visitors,” Joe confirmed.

“Well, I hope you’re prepared for the ride. There’ll be lots of germs and stuff to avoid. How ‘bout I tag along and give you guys some security?”

“If that’s fine with you, then sure!” Mark replied.

Red suddenly tensed. “Contraction’s about to begin. Hold on!”

The room squeezed and the group plunged into the circulation.

Arteries

How does the blood deliver its oxygen? It doesn’t just drop out of the left ventricle and bounce back up. No, it needs a pathway. Some kind of transportation………

The answer is right in the thick-walled blood vessels otherwise known as arteries. These bright red tubes carry oxygenated blood away from the heart and feed the capillaries.

The beginning of all arteries—or as some people call it, the main artery—is called the aorta. The left ventricle pumps blood directly into this major artery, which splits into all of the red strands that you see in really complex pictures.

Another well-known artery is the pulmonary artery. It is unique because not only is it not in the mishmash of the other arteries and veins, it carries deoxygenated blood! The pulmonary artery receives blood from the right ventricle to send to the lungs.

To move blood, the arteries expand to fill up, then shrink down, propelling the blood forward. This is why you feel a pulse when you touch your wrist. The artery there is moving.

Every artery is important in a way, and if you had one more or one less, it would send everything into chaos.


 

Chapter 8

The Comeback

The group whizzed this way and that, turning and diving and ascending everywhere. Bob and Joe whooped as they tilted and spun. Eventually, Red slowed down, allowing the trio to catch their breath. Delta circled protectively around them, keeping watch.

“Wow!” Bob exclaimed. “I totally see why you never get tired of this job.”

“Do you turn blue once you pass through the oxygen-diffusing thingies?” Joe asked.

“Yes,” Red answered. “It’s because of the oxygen-filled hemoglobin that makes me red.”

“I just thought of something.” Everyone turned towards Mark. “We just escaped from the Contaminators. Why haven’t—“

“The Contaminators?!” Red and Delta said in unison.

“Is something wrong?” Mark asked hesitantly.

“The Contaminators are our mortal enemies. We have a history that goes a long way back,” Delta explained. “A long time ago, during the making of the first bodies, the Contaminators wanted to Be part of them, but were rejected by the immune system because of their bacteria. Since then, they have devoted their race to infecting humans, and can double very quickly.”

“Wow, that’s a story worth writing in some dusty old history book!” Joe exclaimed. He suddenly remembered something. “Wait. Shouldn’t the Contaminators be coming after us?”

Suddenly, a large group of white blood cells rushed back against the flow of plasma. One yelled, “The Contaminators are back! Guard the vessels!”

If white blood cells could turn even more pale, Mark could have sworn that Delta had. He pushed the group forward. “Run! They’re coming!”

The group rushed forward, passed through an arteriole, and entered the diffusion site.

Capillaries

The oxygen from our blood has to go somewhere. If it just went around and around the body, what would be the point? The oxygen has to enter our cells to produce energy. But how does it do that?

This where the capillaries come in. Capillaries are tiny, so the oxygenated erythrocytes get close to the walls of these blood vessels, releasing the oxygen while collecting the carbon dioxide.

The smallest capillaries are so tiny that blood cells need to travel through one by one to get across. That is really saying something, because the largest cells are only as wide as the width of one hair.

The capillaries branch all over the body to every cell, so as expected, they are extremely plentiful and are everywhere. They go on for a bit and then feed into the venules. The capillaries don’t stop to wait for the next wave of blood, because it’s constantly coming.

Our capillaries are one of the most important parts of the circulation, and we should be thankful to have them.


 

Chapter 9

Labyrinth

Red weaved through the capillaries, slowly fading to a dark blue. The trio held on to their laser guns, anticipating an attack.

“Gosh, this place is confusing!” Joe remarked.

“Do you think they stopped them?” Bob whispered to Delta.

“Only time will tell,” he replied.

A rumble sounded from behind. The Contaminators and white blood cells battled, getting closer.

“Probably not,” Bob confirmed.

Mark, Joe and Bob helped the leukocytes, shooting lasers everywhere. Still, the Contaminators approached, infecting white blood cells as they went.

Soon, it became clear that they were losing. Mark took charge. “Everybody, stop fighting! Retreat!”

All of the cells immediately turned and flowed after the group, turning this way and that, trying to get away. The trio kept shooting as Red sped out of the capillaries and took the path back to the Alpha.

Veins

Alright, the oxygen has traveled through the arteries and diffused out of the capillaries. What now?

Well, the deoxygenated blood needs to return to the heart to oxygenate again. How does it return? Through the veins.

The venules will join together eventually to form veins, which will take the deoxygenated blood back to the heart. The veins work similarly to the arteries, but they have thinner walls, so they require the help of valves to prevent blood from going back.

Some veins cannot do as well as the others, because gravity is working against them and attempting to have the blood flow down, in the wrong direction. Therefore, the veins with these problems need the help of contracting muscles to force the blood up.

The deoxygenated blood in the veins is actually blue! The reason why you see red blood after bursting a vein is because the blood is hitting the oxygen outside, and so turns red again.

Every part of the cardiovascular system does its part, and the veins are no exception.


 

Chapter 10

The Speed of Light

“We have to get back to the Alpha!” Delta panicked. “She’ll know what to do!”

“We’ve lost so many already.” Red accelerated, going as fast as he could without losing control.

“Keep firing!” Mark yelled as he switched his mode to Electric. The sizzling of electrocution now accompanied the clicks and booms.

A Contaminator smashed open a vein. Platelets rushed to block the blood flow.

Suddenly, Red stopped. The trio lurched backward. “Why are you stopping?!?!?!?!?!?!?” Mark screamed as politely as possible, which wasn’t that polite at all.

“The valve is closed!” Red circled around. “We have to wait for it to open!”

The vein expanded. The Contaminators boosted forward, and………

Commercial Break!

Now, where was I? Oh yeah, right.

The Contaminators boosted forward, and………

The valve burst open, and Red dashed through, completely forgetting about safety. He just thought about going faster, and faster, and faster………

An sonic boom echoed through the vessel, and a swirling purple circle loomed ahead, sucking the trio inside.

?                    ?                    ?

The trio bounced off the floor, coming to a stop against an artery. They got up, examining their surroundings.

“Hey, wait a minute!” Joe exclaimed. “I know this place! We’re in a kidney!”

“Hmmmmmm, that sonic boom must have teleported us here,” Mark observed.

Suddenly, the portal opened again, and the Contaminators spilled out of the gaping hole.

“Get over here!” Bob yelled. He and Joe ran over to the renal artery.

Mark stood his ground. “I’m tired of running.” He drew his gun again. “We make our last stand here.”

The Contaminators charged.

Kidneys

Down there in the urinary system, the kidneys are hard at work, filtering the blood and collecting the waste particles every 25 minutes. They will use the waste to make urine and transport it to the bladder to be execrated.

If you’ve ever wanted to know how big a kidney is, just look at your computer mouse. That is pretty much the size of a kidney—5 inches long and 3 inches wide.

Apart from making urine, kidneys send hormones, like messages, to the bone marrow, telling it to make red blood cells. This is odd for a kidney to do, in my opinion, but is necessary.

My friend, Andre, once said, “Your kidneys look like beans.” This is accurate, of course, but doesn’t it sound wrong?

You could actually live with just one kidney, but that single organ would have to work overtime to keep balancing the water that goes in and out. This is why each person has two kidneys—They make life easier.

The kidneys never get an honorable mention or something, and definitely never get spotted on Valentine’s Day cards, but are important nonetheless.


 

Chapter 11

War

Mark jumped this way and that, firing whenever there was an opening. The Contaminators surged forward, but were always greeted by explosions and sparks.

“I’m not leaving Mark,” Joe declared. He swung his grappling hook, then launched it towards the back of the crowd, where it latched on to a blood vessel. Joe swung down to the battle, and started shooting from the back. The Contaminators, now being attacked from both sides, split into two groups to face the new threat, but it was clear that they were not expecting this.

Bob surveyed the scene as his split personalities fought inside. He pressed his fingers to his temples, feeling a headache coming on. Finally, one side gave in, and he jumped down to join the fight.

Mark glugged down a potion, and immediately enlarged to a much bigger size. He stomped through the continuously splitting creatures, firing his equally large gun. The Contaminators’ numbers started to go down.

Joe swung his grappling hook into the crowd, destroying many of the Contaminators before they had the chances to split. Some of the green creatures started to fight more hesitantly.

Bob quickly opened a bag of popcorn and chugged down the contents. This was not very useful, but he was now not hungry anymore, and the Contaminators he was fighting stopped and looked confused, allowing Mark to squash them from behind.

Eventually, Mark started to shrink down back to normal size, but by then there was only one Contaminator left, the leader. Mark strode forward. “Any last words?”

The Contaminator remained silent, until he spoke one word: “Infection.” He lunged at Mark, knocking him over, and the two battled head to head on the ground.

The leader of the Contaminators produced a bright green fist, and wound up………

…just as a chocolate chip cookie hit him on the head. The Contaminator looked back at Bob’s outstretched hand, momentarily distracted.

Mark shot him in the chest.

The last Contaminator erupted, vibrating the entire kidney. Green bits sprayed around the chamber, dotting the walls of the organ. Bob picked up the cookie and ate it.

Mark stood up slowly, looking around, and relaxed. He took another bottle of shimmering liquid out of his tool belt, and sprinkled a few drops onto a green blotch on his arm. Instantly, it healed.

“Thanks for the cookie, Bob,” Mark thanked. “Never in my life had I thought that bringing lots of snacks would have helped.”

“You’re welcome,” Bob said modestly, but from the look on his face, Mark and Joe could both tell that he was really proud of himself.

“We should probably leave soon,” Mark said. “The kidney’s about to begin filtering.”

“Without saying goodbye to Red and Delta?” Bob asked.

“Of course,” Mark replied. He took out a small orb. He spoke into it. “Hey, Red? Delta? We have to go, like right now, but we wanted to say goodbye first. We stopped the Contaminators. The cells are safe. Have a good life. We might visit some other time, but for now, we must leave. See you next time. End of transmission.” He released the orb. The trio watched it float up into the portal, and watched the portal close.

Mark closed his eyes, as if imagining the adventure again. He looked up, and gave his final command: “Drink your escape potions.”

All three kids uncorked their potions and drank. The group slowly faded out of Steve. The adventure was over.

The End

Posted in Jonathan | Leave a comment

RMAN Backup Failed without any Error messages

In the morning, via OEM Grid Control, I noticed that one of scheduled RMAN backup for production database failed. However, I just saw message like below with our error.

Starting backup at 16-APR-15
channel oem_disk_backup: starting compressed incremental level 1 datafile backup set
input datafile file number=00009 name=/dbFolder/datafile1.dbf
input datafile file number=00003 name=/db_undo_Folder/orcl/undotbs.dbf
input datafile file number=00008 name=/dbFolder/datafile2.dbf
channel oem_disk_backup: specifying datafile(s) in backup set
channel oem_disk_backup: starting piece 1 at 16-APR-15

To get more details, I issued the command below to try to get more information.

select
  j.session_recid, j.session_stamp,
  to_char(j.start_time, 'yyyy-mm-dd hh24:mi:ss') start_time,
  to_char(j.end_time, 'yyyy-mm-dd hh24:mi:ss') end_time,
  (j.output_bytes/1024/1024) output_mbytes, j.status, j.input_type,
  decode(to_char(j.start_time, 'd'), 1, 'Sunday', 2, 'Monday',
                                     3, 'Tuesday', 4, 'Wednesday',
                                     5, 'Thursday', 6, 'Friday',
                                     7, 'Saturday') dow,
  j.elapsed_seconds, j.time_taken_display,
  x.cf, x.df, x.i0, x.i1, x.l,
  ro.inst_id output_instance
from V$RMAN_BACKUP_JOB_DETAILS j
  left outer join (select
                     d.session_recid, d.session_stamp,
                     sum(case when d.controlfile_included = 'YES' then d.pieces else 0 end) CF,
                     sum(case when d.controlfile_included = 'NO'
                               and d.backup_type||d.incremental_level = 'D' then d.pieces else 0 end) DF,
                     sum(case when d.backup_type||d.incremental_level = 'D0' then d.pieces else 0 end) I0,
                     sum(case when d.backup_type||d.incremental_level = 'I1' then d.pieces else 0 end) I1,
                     sum(case when d.backup_type = 'L' then d.pieces else 0 end) L
                   from
                     V$BACKUP_SET_DETAILS d
                     join V$BACKUP_SET s on s.set_stamp = d.set_stamp and s.set_count = d.set_count
                   where s.input_file_scan_only = 'NO'
                   group by d.session_recid, d.session_stamp) x
    on x.session_recid = j.session_recid and x.session_stamp = j.session_stamp
  left outer join (select o.session_recid, o.session_stamp, min(inst_id) inst_id
                   from GV$RMAN_OUTPUT o
                   group by o.session_recid, o.session_stamp)
    ro on ro.session_recid = j.session_recid and ro.session_stamp = j.session_stamp
where j.start_time > trunc(sysdate)-&NUMBER_OF_DAYS
order by j.start_time;

This query generated information below.

image

However, it didn’t provide any further information other than OEM.

And then, I tried to check if all of datafiles are backed up successfully.

SQL> select df.file_name, to_char(bd.completion_time,'YYYY-MM-DD HH24:MI:SS')
from   dba_data_files df
left outer join v$backup_datafile bd
on    (df.file_id = bd.file#)
where bd.completion_time <= sysdate - 1;

FILE_NAME	            TO_CHAR(BD.COMPLETION_TIME 'YYYY-MM-DDHH24:MI:SS')
/dbFolder/system.dbf	                                       4/15/2015 23:48
/dbFolder/sysaux.dbf	                                       4/15/2015 23:52
/db_undo_Folder/undotbs.dbf	                                   4/15/2015 23:40
/dbFolder/datafile1.dbf	                                       4/15/2015 23:50
/dbFolder/datafile2.dbf	                                       4/15/2015 23:50
/dbFolder/datafile3.dbf	                                       4/15/2015 23:50
/dbFolder/datafile4.dbf	                                       4/15/2015 23:49
/dbFolder/datafile5.dbf	                                       4/15/2015 23:43
/dbFolder/datafile6.dbf	                                       4/15/2015 23:48
/dbFolder/datafile7.dbf	                                       4/15/2015 23:50
/dbFolder/datafile8.dbf	                                       4/15/2015 23:49
/dbFolder/datafile9.dbf	                                       4/15/2015 23:50
/dbFolder/datafile10.dbf	                                   4/15/2015 23:49
/dbFolder/datafile11.dbf	                                   4/15/2015 23:50
/dbFolder/datafile12.dbf	                                   4/15/2015 23:50
/dbFolder/datafile13.dbf	                                   4/15/2015 23:50

It’s now clear that datafiles were not backed up for some reasons we don’t know.

I just simply re-run the RMAN job and get success with no problem. And, onwards daily backup got success as well.

Posted in My Reference, Oracle Case Study, Oracle Point | Tagged , | Leave a comment