Analytics


Google

Saturday, May 31, 2008

Automating the Dropping of Oracle Partition

In my previous post, I described how we can automate the creation of Oracle Partition. This is how you can automate the dropping of the partitions.

This code makes the following assumption:
  • Partition is by date
  • The first partition is the table name postfix with zero (e.g. if table is testList and then the name of the first partition should be testList0).
  • Subsequent partition should have the name of the table postfix with _ and the a number (e.g. testList_0, testList_1 ...)

Here is the code:

Imports System.Xml
Imports System.Data.OleDb

' Drop Partition where the partition key is a date field
' Author Strovek
' Date Jul 21, 2005
' Revised Jul 25, 2005
'
' =============================================================================

Public Class DropPartDate

Const DateKey As String = "DateKey"
Const StrKey As String = "StrKey"
Const InvalidKey As String = "InvKey"
Const pfxStr As String = "SELECT PARTITION_NAME, HIGH_VALUE, " _
& "TO_NUMBER(SUBSTR(PARTITION_NAME, LENGTH(TABLE_NAME)+2)) PARTSEQ FROM USER_TAB_PARTITIONS WHERE TABLE_NAME = '"
Const lineStr As String = "================================"

Public Shared generalSwitch As New TraceSwitch("TraceLevelSwitch", "Entire Application")
Dim cfgFile As String = "..\DropPartDate.xml"
Dim Conn As New OleDb.OleDbConnection
Dim connStr As String
Dim cfg As New XmlDocument
Dim root As XmlNode
Dim tabCollection As XmlNodeList
Dim keyType As String
Dim vConfigFound As Boolean = False

' Returns whether the xml configuration is found
Public Function cfgFound() As Boolean
Return vConfigFound
End Function

' Returns the version of this assembly
Public Shared Function Ver() As String
With _
System.Diagnostics.FileVersionInfo.GetVersionInfo(System.Reflection.Assembly.GetExecutingAssembly.Location)
Return .FileMajorPart & "." & .FileMinorPart & "." & .FileBuildPart & "." & .FilePrivatePart
End With
End Function

' Property to set the configuration filename
Public Property cfgFName() As String
Get
cfgFName = cfgFile
End Get
Set(ByVal Val As String)
cfgFile = Val
End Set
End Property

' Reads the xml file
Public Sub readXML()
Dim flg As Boolean = False
Dim xNode As XmlNode
cfg.Load(cfgFile)
xNode = cfg.FirstChild
While True
Trace.WriteLineIf(generalSwitch.TraceVerbose, "xnode name is " & xNode.Name & "*")
If xNode.Name <> "config" Then
xNode = xNode.NextSibling
Else
vConfigFound = True
Exit While
End If
If IsNothing(xNode) Then
Exit While
End If
End While
root = xNode
End Sub

' Displays the Connection String from the xml file
Public Sub getConnStr()
Dim xNode As XmlNode
'xNode = cfg.SelectSingleNode("connStr")
xNode = root.SelectSingleNode("connStr")
Trace.WriteLineIf(generalSwitch.TraceVerbose, xNode.Attributes("Val").Value())
End Sub

' Gets the collection of Tables to process
Public Sub getTabs()
tabCollection = root.SelectNodes("tab")
End Sub

' Display the list of tables to be processed
Public Sub listTabs()
Dim cnt As Integer

For cnt = 0 To (tabCollection.Count - 1)
Console.Write(tabCollection(cnt).Attributes("Name").Value & " ")
Console.WriteLine(tabCollection(cnt).Attributes("Period").Value)
Next

End Sub

' Connect to database
Public Sub ConnectDB()
Dim connStr As String = root.SelectSingleNode("connStr").Attributes("Val").Value
Conn.ConnectionString = connStr
Conn.Open()
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Database connected")
End Sub

' Closes the database connection if it is Open
Public Sub closeDB()
If Conn.State = ConnectionState.Open Then
Conn.Close()
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Database connection Closed")
End If
End Sub

' Verify the key type for the table
Sub chkKeyType(ByVal tabName As String)
Dim dbCmd As OleDbCommand = Conn.CreateCommand
Dim dr As OleDbDataReader
Dim sql, tmpStr, tmpStr2 As String
Dim tmpDate As DateTime
keyType = InvalidKey

sql = pfxStr & tabName & "' and partition_name = '" & tabName & "0'"
dbCmd.CommandText = sql
Try
dr = dbCmd.ExecuteReader
If dr.HasRows Then
dr.Read()
tmpStr = dr.Item("high_value")
If Left(tmpStr, 7) = "TO_DATE" Then
keyType = DateKey
Else
Try
tmpStr2 = tmpStr.Substring(1, 8)
tmpDate = Date.ParseExact(tmpStr2, "yyyyMMdd", Nothing)
keyType = StrKey
Catch ex As Exception
keyType = InvalidKey
End Try
End If
End If
Finally
If Not IsNothing(dr) Then
dr.Close()
End If
End Try

dbCmd.Dispose()
End Sub

' Extract the date from the highvalue field and return it as datetime
Function getDate(ByVal highValue As String) As DateTime
Dim tmpStr As String
Dim tmpDate As DateTime

tmpStr = highValue.Substring(highValue.IndexOf("'") + 2, 19)
Trace.WriteLineIf(generalSwitch.TraceVerbose, tmpStr)
tmpDate = DateTime.ParseExact(tmpStr, "yyyy-MM-dd HH:mm:ss", Nothing)

Return tmpDate.Date
End Function


' Decide which method to extract the date portion of the date from the highvalue column
Function getPartDate(ByVal highval As String) As DateTime
Dim tmpDate As DateTime
Dim tmpStr As String

If keyType = DateKey Then
tmpDate = getDate(highval)
Else
tmpStr = highval.Substring(1, 8)
tmpDate = DateTime.ParseExact(tmpStr, "yyyyMMdd", Nothing)
End If
Return tmpDate.Date
End Function

' Routine to process all the tables
Public Sub procTab()
Dim loopCnt, dropCnt, daysAhead As Integer
Dim sql, tabName, tmpStr, partName As String
Dim partRec As clsPart
Dim dbCmd As OleDbCommand = Conn.CreateCommand
Dim dr As OleDbDataReader
Dim partDate, limDate As DateTime
Dim curDate As DateTime = Now
Dim dropPartList As New ArrayList


For loopCnt = 0 To (tabCollection.Count - 1)
tabName = UCase(tabCollection(loopCnt).Attributes("Name").Value)
daysAhead = tabCollection(loopCnt).Attributes("Period").Value
Console.WriteLine(vbNewLine & "Table " & tabName)
dropPartList.Clear()

chkKeyType(tabName)

If keyType <> InvalidKey Then
limDate = curDate.AddDays(daysAhead).Date
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Date Limit is " & Format(limDate, "dd-MMM-yyyy HH:mm"))

sql = pfxStr & tabName & "' and partition_name not like '" & tabName & "0' order by " _
& "partseq"
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Get partition list sql is " & vbNewLine & sql)
Try
dbCmd.CommandText = sql
dr = dbCmd.ExecuteReader
If dr.HasRows Then
While dr.Read
tmpStr = dr.Item("high_value")
partDate = getPartDate(tmpStr)
partName = dr.Item("partition_name")
partRec = New clsPart
partRec.Name = partName
partRec.LimDate = partDate
Trace.WriteLineIf(generalSwitch.TraceVerbose, partName & " " & Format(partDate, "dd-MMM-yyyy"))
If DateTime.Compare(partDate, limDate) < 0 Then
dropPartList.Add(partRec)
Trace.WriteLineIf(generalSwitch.TraceVerbose, partRec.Name & " " & Format(partRec.LimDate, "dd-MMM-yyyy") & " should be dropped")
Else
Trace.WriteLineIf(generalSwitch.TraceVerbose, partRec.Name & " " & Format(partRec.LimDate, "dd-MMM-yyyy") & " does not qualify")
End If
End While
End If

Finally
If Not IsNothing(dr) Then
dr.Close()
End If
End Try

' If there are partitions that qualify for dropping
If dropPartList.Count > 0 Then
Console.WriteLine(vbNewLine & "Date Limit is " & Format(limDate, "dd-MMM-yyyy HH:mm"))
For dropCnt = 0 To (dropPartList.Count - 1)
partRec = dropPartList(dropCnt)
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Dropping " & partRec.Name & " with " & Format(partRec.LimDate, "dd-MMM-yyyy"))
sql = "alter table " & tabName & " drop partition " & partRec.Name
dbCmd.CommandText = sql
dbCmd.ExecuteNonQuery()
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Drop partition sql is " & vbNewLine & sql)
Console.WriteLine("Dropped " & partRec.Name & " with " & Format(partRec.LimDate, "dd-MMM-yyyy"))
Next
Else
Console.WriteLine("No partitions qualify")
End If
Else
Console.WriteLine(tabName & " is an invalid table")
End If
Next
dbCmd.Dispose()

End Sub

Public Shared Sub Main(ByVal args() As String)
Dim dPart As New DropPartDate
Dim tmpDate As DateTime = Now()
Dim elapseTime As TimeSpan

Console.WriteLine(vbNewLine & "Create Partition (Date) - Version " & dPart.Ver & vbNewLine & lineStr)
Console.WriteLine(Format(Now(), "dd-MMM-yyyy HH:mm"))

Dim writer As TextWriterTraceListener = New TextWriterTraceListener(System.Console.Out)
System.Diagnostics.Debug.Listeners.Add(writer)

If generalSwitch.TraceVerbose Then
Console.WriteLine("Trace is verbose")
End If

Try
' Checks to see if a parameter is provided
If args.Length > 0 Then

' If parameter is ? then display the help
If args(0) = "?" Then
Console.WriteLine("Usage: " & vbNewLine & "crePartDate <configFileName>")
Exit Sub
Else
dPart.cfgFile = args(0)
End If
End If

dPart.readXML()
If dPart.cfgFound Then
dPart.getTabs()

' List table only if trace is verbose
If generalSwitch.TraceVerbose Then
dPart.listTabs()
End If

dPart.ConnectDB()
dPart.procTab()
End If

Catch ex As Exception
Console.WriteLine(vbNewLine & "Error " & vbNewLine & ex.Message)
Trace.WriteIf(generalSwitch.TraceVerbose, ex.StackTrace)
Finally
dPart.closeDB()
End Try

elapseTime = Now.Subtract(tmpDate)
Console.WriteLine(vbNewLine & "Elapse time " & elapseTime.Seconds & " seconds")
Console.WriteLine("End time " & Format(Now(), "dd-MMM-yyyy HH:mm"))
Console.WriteLine("End Program" & vbNewLine & lineStr)
End Sub

End Class

Class clsPart
Dim vName As String
Dim vLim As Date

Public Property Name() As String
Get
Return vName
End Get
Set(ByVal Value As String)
vName = Value
End Set
End Property

Public Property LimDate() As Date
Get
Return vLim
End Get
Set(ByVal Value As Date)
vLim = Value
End Set
End Property
End Class

As in creating partition, I also use an xml instead of the config file. This is to allow me to drop partition from multiple database instance by supporting each database with a separate xml file. Here is a sample xml file:
<config>
<connStr Val="Provider=ORAOLEDB.Oracle;PASSWORD=tiger;User ID=scott;Data Source=orcl" />
<tab Name="TESTPART" Period="-200" />
<tab Name="TESTPART2" Period="-300" />
<tab Name="MFGTRANS" Period="-250" />
<tab Name="PARTMASTER" Period="-200" />
</config>

Monday, May 26, 2008

Log Parser

After starting to use the new log feature in Framework 2.x, how do you analyze the log file. There is a free tool provided by Microsoft to do just that. That tool can be obtained from here. However, it can be quite challenging since it is a line command utility. Also in order for it to work, you need to make sure you use , as the delimiter so that it can treat your log file as a CSV file.

Luckily somebody else had taken an initiative to make it better by coming up with a GUI interface called Visual LogParser, unfortunately, I found this to be lacking. It did not allow some of the options.

You can read more about the features of LogParser from Microsoft, or from another blogger here who has also linked some other resources.

Sunday, May 25, 2008

Enhanced Tracing in Framework 2.0

Following up on the previous post on logging, seems that the same mechanism can be used on the tracing. In the case of application with multiple modules, you can create multiple TraceSource and associate each TraceSource to each module. That way, you can trace output from each module separately.

To activate the tracesource, you just have to retrieve the tracesource in your program as follows:

Dim ts as new TraceSource("MainForm")

"MainForm" is the name of the tracesource as seen in the config file:
  <system.diagnostics>
<sources>
<!-- This section defines the logging configuration for My.Application.Log -->
<source name="MainForm" switchValue="All">
<listeners>
<add name="LogToFile"/>
</listeners>
</source>
</sources>
<switches>
<add name="DefaultSwitch" value="All" />
<add name="ActSwitch" value="ActivityTracing" />
<add name="VerSwitch" value="Verbose" />
</switches>
<sharedListeners>
<!-- add name="FileLog"
type="Microsoft.VisualBasic.Logging.FileLogTraceListener, Microsoft.VisualBasic, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL"
initializeData="FileLogWriter"/ -->
<!-- Uncomment the below section and replace APPLICATION_NAME with the name of your application to write to the Application Event Log -->
<!--<add name="EventLog" type="System.Diagnostics.EventLogTraceListener" initializeData="APPLICATION_NAME"/> -->
<add name="LogToFile" type="Microsoft.VisualBasic.Logging.FileLogTraceListener, Microsoft.VisualBasic, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL"
initializeData="FileLogWriter"
logFileCreationSchedule="Daily"
traceOutputOptions="DateTime"
BaseFileName="MainWin"
location="Custom"
CustomLocation="c:\log\"
Delimiter ="|"
AutoFlush="True"
Append="True"
/>
</sharedListeners>
</system.diagnostics>


This is superior to the one available in Framework 1.1 because, instead of using a If statement to check the tracelevel prior to performing Trace.WriteLine, you use ts.TraceEvent(TraceEventType.Information, 1, "Started Trace").

In framework 1.1, I facilitated this by creating a subroutine:

'==================================================
' performs a trace write if the switch is > then the pLevel
' If flg = 0 then do a trace.write
' if flg = 1 then do a trace.writeline
' =================================================
Sub traceMsg(ByRef tSwitch As TraceSwitch, ByVal pLevel As TraceLevel, _
ByVal pMsg As String, Optional ByVal traceCat As String = "Debug", _
Optional ByVal flg As Integer = 0)

If tSwitch.Level >= pLevel Then
If flg = 0 Then
Trace.Write(traceCat, pMsg)
Else
Trace.WriteLine(traceCat, pMsg)
End If

End If
End Sub

Application Logging

Some new classes were introduced to facilitate application logging. Previously, we had to depend on 3rd party libraries like log4net. One of the classes that are most useful is the FileLogTraceListener which provided autoarchiving by date or by application.

I started exploring the information using information from the this article.

First lets start with the app.config file:

<system.diagnostics>
<sources>
<!-- This section defines the logging configuration for My.Application.Log -->
<source name="DefaultSource" switchName="MSwitch">
<listeners>
<add name="LogToFile" />
</listeners>
</source>
</sources>
<switches>
<add name="MSwitch" value="All" />
</switches>
<sharedListeners>
<add name="FileLog"
type="Microsoft.VisualBasic.Logging.FileLogTraceListener, Microsoft.VisualBasic, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL"
initializeData="FileLogWriter"/>
<!-- Uncomment the below section and replace APPLICATION_NAME with the name of your application to write to the Application Event Log -->
<!--<add name="EventLog" type="System.Diagnostics.EventLogTraceListener" initializeData="APPLICATION_NAME"/> -->
<add name="LogToFile" type="Microsoft.VisualBasic.Logging.FileLogTraceListener, Microsoft.VisualBasic, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL"
initializeData="FileLogWriter"
logFileCreationSchedule="Daily"
traceOutputOptions="DateTime"
BaseFileName="W_Test"
location="Custom"
CustomLocation="c:\log\"
Delimiter ="|"
AutoFlush="True"
Append="True"
/>
</sharedListeners>
</system.diagnostics>

To use be able to use My.Application.log class, you need a source called DefaultSource, you can then point it to a switch name. In the switch name (or place the value directly using SwitchValue, you have an option to identify the level of logging (taken from the help file) :

ActivityTracing Enumeration
Description
Off Does not allow any events through.
Critical Allows only Critical events through.
Error Allows Critical and Error events through.
Warning Allows Critical, Error, and Warning events through.
Information Allows Critical, Error, Warning, and Information events through.
Verbose Allows Critical, Error, Warning, Information, and Verbose events through.
ActivityTracing Allows the Stop, Start, Suspend, Transfer, and Resume events through.
All Allows all events through.


You can then add various listeners. If you are using FileLogTraceListener,
  • you can specify the BaseFileName. This will be the used as your log file name.
  • specify whether or not to append to the log.
  • autoflush
  • location (if specify Custom then specify CustomLocation to tell it where to put the file).
  • logFileCreationSchedule whether to create Daily, Weekly or None.
  • Delimiter (character to use as delimiter between the columns in the log file).
  • TraceOutputOption (additional items to automatically add to the log file). If you use DateTime, it will provide it give you the time in GMT timezone.
Based on the above configuration, my log file will have the name W_Test-2008-05-24.log and the content will look something like this:

DefaultSource|Start|4|Entering Button2_click|2008-05-24 03:37:54Z
DefaultSource|Stop|5|Leaving Button2_click|2008-05-24 03:37:54Z
DefaultSource|Information|0|Going thru Button2|2008-05-24 03:37:54Z

Saturday, May 24, 2008

Changing Color in DataGridView based on content

The following is how it can be achieved:

Private Sub dgView_RowPostPaint(ByVal sender As Object, ByVal e As System.Windows.Forms.DataGridViewRowPostPaintEventArgs) Handles dgView.RowPostPaint
Dim vValue As String = ""
Dim vCnt As Integer = dgView.Rows.Count
Dim cellCnt As Integer

If Me.Visible Then
If vCnt > 0 Then
vValue = dgView.Rows(e.RowIndex).Cells(3).Value
If IsNothing(vValue) = False Then
If LCase(vValue).Contains("err") Then
For cellCnt = 0 To (dgView.Rows(e.RowIndex).Cells.Count - 1)
dgView.Rows(e.RowIndex).Cells(cellCnt).Style.ForeColor = Color.Red
Next
End If
End If
End If

End If
End Sub



In this example, I am looking at the fourth column and if it contains the word "err", the words will be turned red.



An earlier routine using RowsAddEvent did not work when I converted from an unsorted data to sorted data.

Friday, May 23, 2008

Removing Close Button From Windows

This snippet was obtained from a reply found here.

Private Declare Auto Function GetSystemMenu Lib "user32.dll" (ByVal hWnd As IntPtr, _
ByVal bRevert As Int32 _
) As IntPtr

Private Declare Auto Function GetMenuItemCount Lib "user32.dll" ( _
ByVal hMenu As IntPtr _
) As Int32

Private Declare Auto Function DrawMenuBar Lib "user32.dll" ( _
ByVal hWnd As IntPtr _
) As Int32

Private Declare Auto Function RemoveMenu Lib "user32.dll" ( _
ByVal hMenu As IntPtr, _
ByVal nPosition As Int32, _
ByVal wFlags As Int32 _
) As Int32

Private Const MF_BYPOSITION As Int32 = &H400
Private Const MF_REMOVE As Int32 = &H1000

Private Sub RemoveCloseButton(ByVal frmForm As Form)
Dim hMenu As IntPtr, n As Int32
hMenu = GetSystemMenu(frmForm.Handle, 0)
If Not hMenu.Equals(IntPtr.Zero) Then
n = GetMenuItemCount(hMenu)
If n > 0 Then
RemoveMenu(hMenu, n - 1, MF_BYPOSITION Or MF_REMOVE)
RemoveMenu(hMenu, n - 2, MF_BYPOSITION Or MF_REMOVE)
DrawMenuBar(frmForm.Handle)
End If
End If
End Sub


The other code found here doesn't work. It grays out the X button but it is still functional.

Monday, May 19, 2008

Application Scheduling

Sometimes it is necessary to perform activities on regular intervals such as:
  1. Sending daily email alert.
  2. Compiling wip @ a specific time.
  3. Compiling yield or transactions either shiftly or daily.
This can be easily done by writing a console (non GUI) application and then scheduling the application to run at regular intervals. With Windows 2000, Windows XP, and Windows 2003, you can use the Windows scheduler.

The Windows scheduler was first introduced in Windows NT but it did not provide a nice interface for adding to and viewing the schedule (using AT command - similar to the command available in Unix). This has since been improved.

Nice thing about the Windows Scheduler is you can set it to run as different users. Typically, most will run under Admin account. However, note that when it runs, the screen will only display in the console.

One of the benefit, Windows Scheduler has over the Unix (cronjob) is that it has a built-in mechanism that will ensure only a copy of the schedule item will run at any time. While Unix does not do that. For example, you may schedule extractData and schedule it to run every 3 minutes. In Unix, if the extract takes more than 3 minutes, you may end up with multiple copies running at the same time. This will not happen if scheduled in Window scheduler.

There are also some failsafe such as automatically abort if the job takes more than a specified amount of time.

Finally when we schedule a job we typically expect it to run automatically. There are some additional precaution we should take:
  • have a log file output so that we can trace when there are problems. Easiest way is to redirect the output of the program (for example extractData >> c:\log\extractData.log)
  • Have the application generate a heartbeat file each time it completes successfully and then have another program monitor the heartbeat file modified time. If the heartbeat update time exceeds a threshold, just have the application send an alert.

Saturday, May 17, 2008

Automating Creation of Oracle Partition

A typical way of creating Oracle partition is to partition by a column representing a date field. However, it is possible to represent the date either by a date type or as Varchar2. When representing the date as Varchar2, we normally represent it as "yyyymmdd".

Let me share with you a vb.net console program I wrote to automate the creation of partition.


First lets create two tables:

CREATE TABLE TESTPART
(RECID NUMBER,
LOTID VARCHAR2(20),
OLD_LOTID VARCHAR2(20),
ACTIVITY VARCHAR2(40),
TXNTIME CHAR(18),
EXTDATE VARCHAR2(15),
FROMTIME VARCHAR2(18),
TOTIME VARCHAR2(18),
NOTES VARCHAR2(255),
BRIEFDESC VARCHAR2(255),
DETAILDESC VARCHAR2(255)
)
PARTITION BY RANGE (EXTDATE)
(PARTITION TESTPART0 VALUES LESS THAN ('20010101 080001'));


CREATE TABLE TESTPART2
( TIME_DATE DATE,
LOTID VARCHAR2(20),
ACTIVITY VARCHAR2(40),
STEPNAME VARCHAR2(60 BYTE),
REASONCODE VARCHAR2(20 BYTE),
REASONQTY NUMBER,
CATEGORY VARCHAR2(6 BYTE),
BONUSQTY NUMBER
)
PARTITION BY RANGE (TIME_DATE)
(PARTITION FW_WOP_DEF0 VALUES LESS THAN (TO_DATE('20010101', 'YYYYMMDD'));

Both tables are partitioned by date but in the first one the date is stored as varchar2 and in the second one it is stored as a date field.

In order to start the process, we need to know how to get information on the partition. This is available in USER_TAB_PARTITIONS view. Next you need to know how the know how the data is stored. The column High_value, stores high value of the partition key (meaning the max value the partition will store. The problem with this is that the column type is long() and cannot be analyzed directly within PL/SQL which is why I had to use an external program to analyze the value otherwise, I would have created a stored procedure to automate the process.

If the partition key is of date type, then it will be stored as (TO_DATE(' 2001-01-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')). For varchar2, it will be stored as an actual value. This will allow us to check the partition key type that we use to perform the logic whether or not to create the partition.

To facilitate the automation, I follow the following convention:
  • Have a reference partition which is the table name followed by '0' , so in the example above the table is testpart and the reference partition is testpart0.
  • Then when I create the partition, I use a sequential numbering, ie. the partition will be testpart_1, testpart_2 etc.
  • This is used so that when I automate the dropping of partition, I will not accidentally drop the reference partition.
A few notes on the code:
  • ReadXML will read and load the xml into an XMLDocument and point the xmlnode to the right node (<config>).
  • chkKeyType will identify from the high_value whether the type used in the partition key column (varchar2 or Date).
  • getDate will get the highest high_value of the partitions that exist for the table.
  • procTab will go through the list of table provided in the XML. It will also compare the target date to create the partition and decide whether it is necessary to create the partition. If necessary then it will take the sequence from the last partition increment 1 and then use it as the name to create the new partition.

Finally, here is the code:
Imports System.Xml
Imports System.Data.OleDb

' Create Partition where the partition key is a date field
' Author Strovek
' Date Jul 21, 2005
' Revised
'
' Rewrote the program to recognize the different date partition key.
' date field or string with the format "yyyymmdd hh24miss"
'
' =============================================================================

Public Class CrePartDate

Const DateKey As String = "DateKey"
Const StrKey As String = "StrKey"
Const InvalidKey As String = "InvKey"
Const pfxStr As String = "SELECT PARTITION_NAME, HIGH_VALUE, " _
& "TO_NUMBER(SUBSTR(PARTITION_NAME, LENGTH(TABLE_NAME)+2)) PARTSEQ FROM USER_TAB_PARTITIONS WHERE TABLE_NAME = '"
Const lineStr As String = "================================"

Public Shared generalSwitch As New TraceSwitch("TraceLevelSwitch", "Entire Application")
Dim cfgFile As String = "..\crePartDate.xml"
Dim Conn As New OleDb.OleDbConnection
Dim connStr As String
Dim cfg As New XmlDocument
Dim root As XmlNode
Dim tabCollection As XmlNodeList
Dim keyType As String
Dim vConfigFound As Boolean = False

' Returns the version of this assembly
Public Shared Function Ver() As String
With _
System.Diagnostics.FileVersionInfo.GetVersionInfo(System.Reflection.Assembly.GetExecutingAssembly.Location)
Return .FileMajorPart & "." & .FileMinorPart & "." & .FileBuildPart & "." & .FilePrivatePart
End With
End Function

Public Function cfgFound() As Boolean
Return vConfigFound
End Function

' Property to set the configuration filename
Public Property cfgFName() As String
Get
cfgFName = cfgFile
End Get
Set(ByVal Val As String)
cfgFile = Val
End Set
End Property

' Reads the xml file
Public Sub readXML()
Dim flg As Boolean = False
Dim xNode As XmlNode
cfg.Load(cfgFile)
xNode = cfg.FirstChild
While True
Trace.WriteLineIf(generalSwitch.TraceVerbose, "xnode name is " & xNode.Name & "*")
If xNode.Name <> "config" Then
xNode = xNode.NextSibling
Else
vConfigFound = True
Exit While
End If
If IsNothing(xNode) Then
Exit While
End If
End While
root = xNode
End Sub

' Displays the Connection String from the xml file
Public Sub getConnStr()
Dim xNode As XmlNode
'xNode = cfg.SelectSingleNode("connStr")
xNode = root.SelectSingleNode("connStr")
Trace.WriteLineIf(generalSwitch.TraceVerbose, xNode.Attributes("Val").Value())
End Sub

' Gets the collection of Tables to process
Public Sub getTabs()
tabCollection = root.SelectNodes("tab")
End Sub

' Display the list of tables to be processed
Public Sub listTabs()
Dim cnt As Integer

For cnt = 0 To (tabCollection.Count - 1)
Console.Write(tabCollection(cnt).Attributes("Name").Value & " ")
Console.WriteLine(tabCollection(cnt).Attributes("Period").Value)
Next

End Sub

' Connect to database
Public Sub ConnectDB()
Dim connStr As String = root.SelectSingleNode("connStr").Attributes("Val").Value
Conn.ConnectionString = connStr
Conn.Open()
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Database connected")
End Sub

' Closes the database connection if it is Open
Public Sub closeDB()
If Conn.State = ConnectionState.Open Then
Conn.Close()
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Database connection Closed")
End If
End Sub

' Verify the key type for the table
Sub chkKeyType(ByVal tabName As String)
Dim dbCmd As OleDbCommand = Conn.CreateCommand
Dim dr As OleDbDataReader
Dim sql, tmpStr, tmpStr2 As String
Dim tmpDate As DateTime
keyType = InvalidKey

sql = pfxStr & tabName & "' and partition_name = '" & tabName & "0'"
dbCmd.CommandText = sql
Try
dr = dbCmd.ExecuteReader
If dr.HasRows Then
dr.Read()
tmpStr = dr.Item("high_value")
If Left(tmpStr, 7) = "TO_DATE" Then
keyType = DateKey
Else
Try
tmpStr2 = tmpStr.Substring(1, 8)
tmpDate = Date.ParseExact(tmpStr2, "yyyyMMdd", Nothing)
keyType = StrKey
Catch ex As Exception
keyType = InvalidKey
End Try
End If
End If
Finally
If Not IsNothing(dr) Then
dr.Close()
End If
End Try

dbCmd.Dispose()
End Sub

' Extract the date from the highvalue field and return it as datetime
Function getDate(ByVal highValue As String) As DateTime
Dim tmpStr As String
Dim tmpDate As DateTime

tmpStr = highValue.Substring(highValue.IndexOf("'") + 2, 19)
Trace.WriteLineIf(generalSwitch.TraceVerbose, tmpStr)
tmpDate = DateTime.ParseExact(tmpStr, "yyyy-MM-dd HH:mm:ss", Nothing)

Return tmpDate.Date
End Function


' Decide which method to extract the date portion of the date from the highvalue column
Function getPartDate(ByVal highval As String) As DateTime
Dim tmpDate As DateTime
Dim tmpStr As String

If keyType = DateKey Then
tmpDate = getDate(highval)
Else
tmpStr = highval.Substring(1, 8)
tmpDate = DateTime.ParseExact(tmpStr, "yyyyMMdd", Nothing)
End If
Return tmpDate.Date
End Function

' Routine to process all the tables
Public Sub procTab()
Dim loopCnt, daysAhead, partseq As Integer
Dim tabName, tmpStr, partName, sql As String
Dim dbCmd As OleDbCommand = Conn.CreateCommand
Dim dr As OleDbDataReader
Dim partDate, limDate As DateTime
Dim curDate As DateTime = Now()
Dim crePartFlg As Boolean

For loopCnt = 0 To (tabCollection.Count - 1)
tabName = UCase(tabCollection(loopCnt).Attributes("Name").Value)
daysAhead = tabCollection(loopCnt).Attributes("Period").Value
Console.WriteLine(vbNewLine & "Table " & tabName & " ")
chkKeyType(tabName)
If keyType <> InvalidKey Then
crePartFlg = False
If daysAhead > 0 Then

limDate = curDate.AddDays(daysAhead).Date

sql = pfxStr & tabName & "' and partition_name not like '" & tabName & "0' order by " _
& "to_number(substr(partition_name, length(table_name)+2)) desc"
dbCmd.CommandText = sql
Try
dr = dbCmd.ExecuteReader
If dr.HasRows Then
dr.Read()
tmpStr = dr.Item("high_value")
partDate = getPartDate(tmpStr)
Console.WriteLine("Date limit is " & Format(limDate, "dd-MMM-yyyy HH:mm"))
Console.WriteLine("Part Limit is " & Format(partDate, "dd-MMM-yyyy HH:mm"))

If DateTime.Compare(partDate, limDate) < crepartflg =" True" partseq =" dr.Item(" crepartflg =" True" partseq =" 1" partname =" tabName" keytype =" DateKey" sql = "alter table " sql = "alter table " tmpstr = "Create partition script " commandtext =" sql" datetime =" Now()" textwritertracelistener =" New"> 0 Then

' If parameter is ? then display the help
If args(0) = "?" Then
Console.WriteLine("Usage: " & vbNewLine & "crePartDate
")
Exit Sub
Else
cPart.cfgFile = args(0)
End If
End If

cPart.readXML()
If cPart.cfgFound Then
cPart.getTabs()

' List table only if trace is verbose
If generalSwitch.TraceVerbose Then
cPart.listTabs()
End If

cPart.ConnectDB()
cPart.procTab()
End If

Catch ex As Exception
Console.WriteLine(vbNewLine & "Error " & vbNewLine & ex.Message)
Trace.WriteIf(generalSwitch.TraceVerbose, ex.StackTrace)
Finally
cPart.closeDB()
End Try

elapseTime = Now.Subtract(tmpDate)
Console.WriteLine(vbNewLine & "Elapse time " & elapseTime.Seconds & " seconds")
Console.WriteLine("End time " & Format(Now(), "dd-MMM-yyyy HH:mm"))
Console.WriteLine("End Program" & vbNewLine & lineStr)
End Sub

End Class

Since we typically may have multiple database instances or schema, I use an xml instead of the config file so that I can use one xml for each account/database and then have multiple tables in the xml. For example:
<?xml version="1.0" encoding="utf-8" ?>
<!--
Document : crePartDate.xml
Created on : Jul 21, 2005
Modified : Jul 25, 2005
Author : Strovek
Description:
Configuration file for crePart Project
-->
<config>
<connStr Val="Provider=ORAOLEDB.Oracle;PASSWORD=tiger;User ID=scott;Data Source=orcl" />
<tab Name="TESTPART" Period="7" />
<tab Name="TESTPART2" Period="7" />
</config>
<!--
Modification History
-->

The period means how many days ahead I need to create the partition, in the example ahead it is 7 days so if today is May 17, 2008, the program will create a partition with the high_value of May 24, 2008. So to run the program, just execute crePartDate crePartDate.xml. With this you can then use the Windows scheduler to run either daily, weekly etc.

This concept can be done in any other language like C# and java.

Wednesday, May 14, 2008

Interface Matters

One of the key things in programming is the interface. This is how the user interacts with our application. Take example of some of the programs we used to use. In word processing, we had WordStar, then WordPerfect and now Microsoft Word.

Why did Wordstar die off, it is due to the interface? WordPerfect was more intuitive than Wordstar. WordPerfect provided the first WYSIWYG. Then MS Word came along and provide an even more intuitive interface.

Another example is Lotus 1-2-3 again killed off by an application providing a more intuitive interface - namely Excel.

Word of caution though, intuition is very subjective and familiarity comes into play also. For example in some applications like web sites, there are certain expectations like the menu is on the top or side. So we have also got to take into consideration user expectation. Another example is when we drive a car, we expect a steering wheel and not a joystick.

Two years ago, I attended Lotusphere and one of the session covered was "Interface Matters". Go to the website. Even though they use Lotus Domino as a platform, some of the ideas covered can be used in any other platform.

Think of interface as packaging for your software. You can see some fantastic ideas out there such as Ipod, Macintosh. Elegance and simplicity is the key to a successful interface. Interface designing is an art as well as a science so have fun with it but make sure to get feedback from your users!!

Saturday, May 10, 2008

How to WaterMark an Image file

Nowadays, we place a lot of images on Internet which can be easily copied and used by others. One way is to place a watermark on the image so that even if others copy your images, you copyright or whatever note you have will still be in the image.

This is how you can do that using vb.net


Public Sub addWaterMark(ByVal frmFileName As String, ByVal toFileName As String, ByVal pStr As String)
Dim Img As Image
Dim g As Graphics
Dim brsh As Brush

Try
Img = Image.FromFile(frmFileName)
g = Graphics.FromImage(Img)
Dim fontFamily As New FontFamily("Arial")
Dim font As New Font( _
fontFamily, _
60, _
FontStyle.Regular, _
GraphicsUnit.Pixel)
brsh = Brushes.Gray
g.DrawString(pStr, font, Brushes.Gray, 300, 50)
g.DrawImage(Img, 0.0F, 0.0F)
Img.Save(toFileName)
Console.WriteLine("Watermark done")

Catch ex As Exception
Throw New Exception("[addWaterMark01]" & ex.Message)
End Try
End Sub

As shown above,
  1. First load the image file into an image object
  2. Create a Graphics object from the image object.
  3. Then write the message you want, using the font, color and size needed.
  4. Draw the Graphics back into the image.
  5. Then save the image to the new file (so you still have your original file.
If you want you can change it further to process a folder instead of a file so that you can perform the watermarking in batch mode prior to uploading into internet.

Thursday, May 8, 2008

Working in XML in VB.net

The easiest way to create xml from scratch is to use the XMLDocument class.

All XML needs to have a root node. To do that, just declare a new XMLDocument class object and then use it to create the root node.

For example if you want to create the following xml:

<Book>
<Title>Programmer's Journal</Title>
<Author Name="Me" />
</Book>


You can write something like this:

Imports System.Xml

Public Class clsTest

Private Shared Function genXML() As XmlNode
Dim xRoot, xNode As XmlNode
Dim xAttr As XmlAttribute
Dim xDoc As New XmlDocument

xRoot = xDoc.CreateElement("Book")

Try
xNode = xDoc.CreateElement("Title")
xNode.InnerText = "Programmer's Journal"
xRoot.AppendChild(xNode)
xNode = xDoc.CreateElement("Author")
xAttr = xDoc.CreateNode(XmlNodeType.Attribute, "Name", xNode.NamespaceURI)
xAttr.Value = "Me"
xNode.Attributes.Append(xAttr)
xRoot.AppendChild(xNode)

Catch ex As Exception
Throw New Exception("[genXML01]" & ex.Message)
End Try

Return xRoot
End Function

Public Shared Sub main(ByVal args() As String)
Console.WriteLine(clsTest.genXML.OuterXml)
End Sub

End Class

Running the program above will generate an output like this:

<Book><Title>Programmer's Journal</Title><Author Name="Me" /></Book>


Note that you use XMLDocument.CreateElement to create each node and then append subsequent nodes to root node. The values within the tags are call innerText and then you can also get the whole xml by getting the OuterXML property from the root node. Values within the nodes are called attributes and is generated using XMLDocument.CreateNode and specifying the node type.

One gotcha, you can only append XMLNodes that are created from the same XMLDocument as child to the nodes.

Wednesday, May 7, 2008

.Net IDE

When I started work with .Net several years back, I started out with Web Matrix. This only supports Framework 1.1 but it was sufficient to get me working, however, it lacks the intellisense. Then I found other free IDEs that help with my work such as SharpDeveloper. This was better and it provide a limited intellisense capability. SharpDeveloper had its own project files so was in a way similar to Visual Studio. However, the problem I had with it was if I was to add reference to an assembly it memorizes it. So if I was to refer to my custom assembly and then decide to recompile that assembly, it was necessary to go and remove it from the reference.

Snippet Compiler was also a very nice IDE which again provide some limited intellisense. However, when Framework 2.0 was released, majority of the IDEs broke.

For a time, I could not find any good IDE but finally Microsoft came out with the express edition for Visual Studio 2005. Currently there is also a Express edition for Visual Studio 2008. The express edition of Visual Studio provided most of the functionalities of Visual Studio with some limitations.

Main limitations are:
  • Instead of everything in one, you have different packages
    • Web design
    • Windows/Console app (1 for VB and 1 for C# and 1 for C++)
  • WorkFlow capability is removed from VS 2008 express.
However, if you are just looking for a good IDE to help with writing the code then I think it is more then sufficient. I love using the ability to open a folder as a website and then just writing the code. This is useful when I want to do some quick programming without bothering with projects and solutions. You can then use either command prompt to compile or use a tool called NANT to compile your codes.

Thursday, May 1, 2008

DataTable as a Database

Continuing the previous article on offline scallable computing, the we can use datatable as our database.

There are a many methods and classes that provide support this. First we have the DataSet which can be thought as a database containing many DataTables. You can save the dataset directly into an XML file or create a DataSet directly from the XML file.
  • Writing can be done using either DataSet.WriteXml(filePath, XmlWriteMode.WriteSchema).
  • Retrieving the data is back to the DataSet is done using DataSet.ReadXML(filePath).
You can then use DataView to facilitate the selection of data. For example if you have an employee table and you want to select an employee record using the dataview, you could do the following:

DIM dv as DataView

dv = New DataView(ds.Tables("EmpMaster"), "empno='111000'")


You can update the record retrieved in the DataView by performing a BeginEdit and EndEdit.

For example you can update the persons address as follows:

dv.AllowEdit = True 'This is because by default the view is read-only
dv.BeginEdit()
dv(0)("Address") = "111 AprilFool Lane"
dv.EndEdit()

This all happens in memory so after updating, you need to write the dataset back to the file. This is the only weakness of this approach since we will need to write back the whole dataset even though we just updated one column in one row.