Analytics


Google

Saturday, May 31, 2008

Automating the Dropping of Oracle Partition

In my previous post, I described how we can automate the creation of Oracle Partition. This is how you can automate the dropping of the partitions.

This code makes the following assumption:
  • Partition is by date
  • The first partition is the table name postfix with zero (e.g. if table is testList and then the name of the first partition should be testList0).
  • Subsequent partition should have the name of the table postfix with _ and the a number (e.g. testList_0, testList_1 ...)

Here is the code:

Imports System.Xml
Imports System.Data.OleDb

' Drop Partition where the partition key is a date field
' Author Strovek
' Date Jul 21, 2005
' Revised Jul 25, 2005
'
' =============================================================================

Public Class DropPartDate

Const DateKey As String = "DateKey"
Const StrKey As String = "StrKey"
Const InvalidKey As String = "InvKey"
Const pfxStr As String = "SELECT PARTITION_NAME, HIGH_VALUE, " _
& "TO_NUMBER(SUBSTR(PARTITION_NAME, LENGTH(TABLE_NAME)+2)) PARTSEQ FROM USER_TAB_PARTITIONS WHERE TABLE_NAME = '"
Const lineStr As String = "================================"

Public Shared generalSwitch As New TraceSwitch("TraceLevelSwitch", "Entire Application")
Dim cfgFile As String = "..\DropPartDate.xml"
Dim Conn As New OleDb.OleDbConnection
Dim connStr As String
Dim cfg As New XmlDocument
Dim root As XmlNode
Dim tabCollection As XmlNodeList
Dim keyType As String
Dim vConfigFound As Boolean = False

' Returns whether the xml configuration is found
Public Function cfgFound() As Boolean
Return vConfigFound
End Function

' Returns the version of this assembly
Public Shared Function Ver() As String
With _
System.Diagnostics.FileVersionInfo.GetVersionInfo(System.Reflection.Assembly.GetExecutingAssembly.Location)
Return .FileMajorPart & "." & .FileMinorPart & "." & .FileBuildPart & "." & .FilePrivatePart
End With
End Function

' Property to set the configuration filename
Public Property cfgFName() As String
Get
cfgFName = cfgFile
End Get
Set(ByVal Val As String)
cfgFile = Val
End Set
End Property

' Reads the xml file
Public Sub readXML()
Dim flg As Boolean = False
Dim xNode As XmlNode
cfg.Load(cfgFile)
xNode = cfg.FirstChild
While True
Trace.WriteLineIf(generalSwitch.TraceVerbose, "xnode name is " & xNode.Name & "*")
If xNode.Name <> "config" Then
xNode = xNode.NextSibling
Else
vConfigFound = True
Exit While
End If
If IsNothing(xNode) Then
Exit While
End If
End While
root = xNode
End Sub

' Displays the Connection String from the xml file
Public Sub getConnStr()
Dim xNode As XmlNode
'xNode = cfg.SelectSingleNode("connStr")
xNode = root.SelectSingleNode("connStr")
Trace.WriteLineIf(generalSwitch.TraceVerbose, xNode.Attributes("Val").Value())
End Sub

' Gets the collection of Tables to process
Public Sub getTabs()
tabCollection = root.SelectNodes("tab")
End Sub

' Display the list of tables to be processed
Public Sub listTabs()
Dim cnt As Integer

For cnt = 0 To (tabCollection.Count - 1)
Console.Write(tabCollection(cnt).Attributes("Name").Value & " ")
Console.WriteLine(tabCollection(cnt).Attributes("Period").Value)
Next

End Sub

' Connect to database
Public Sub ConnectDB()
Dim connStr As String = root.SelectSingleNode("connStr").Attributes("Val").Value
Conn.ConnectionString = connStr
Conn.Open()
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Database connected")
End Sub

' Closes the database connection if it is Open
Public Sub closeDB()
If Conn.State = ConnectionState.Open Then
Conn.Close()
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Database connection Closed")
End If
End Sub

' Verify the key type for the table
Sub chkKeyType(ByVal tabName As String)
Dim dbCmd As OleDbCommand = Conn.CreateCommand
Dim dr As OleDbDataReader
Dim sql, tmpStr, tmpStr2 As String
Dim tmpDate As DateTime
keyType = InvalidKey

sql = pfxStr & tabName & "' and partition_name = '" & tabName & "0'"
dbCmd.CommandText = sql
Try
dr = dbCmd.ExecuteReader
If dr.HasRows Then
dr.Read()
tmpStr = dr.Item("high_value")
If Left(tmpStr, 7) = "TO_DATE" Then
keyType = DateKey
Else
Try
tmpStr2 = tmpStr.Substring(1, 8)
tmpDate = Date.ParseExact(tmpStr2, "yyyyMMdd", Nothing)
keyType = StrKey
Catch ex As Exception
keyType = InvalidKey
End Try
End If
End If
Finally
If Not IsNothing(dr) Then
dr.Close()
End If
End Try

dbCmd.Dispose()
End Sub

' Extract the date from the highvalue field and return it as datetime
Function getDate(ByVal highValue As String) As DateTime
Dim tmpStr As String
Dim tmpDate As DateTime

tmpStr = highValue.Substring(highValue.IndexOf("'") + 2, 19)
Trace.WriteLineIf(generalSwitch.TraceVerbose, tmpStr)
tmpDate = DateTime.ParseExact(tmpStr, "yyyy-MM-dd HH:mm:ss", Nothing)

Return tmpDate.Date
End Function


' Decide which method to extract the date portion of the date from the highvalue column
Function getPartDate(ByVal highval As String) As DateTime
Dim tmpDate As DateTime
Dim tmpStr As String

If keyType = DateKey Then
tmpDate = getDate(highval)
Else
tmpStr = highval.Substring(1, 8)
tmpDate = DateTime.ParseExact(tmpStr, "yyyyMMdd", Nothing)
End If
Return tmpDate.Date
End Function

' Routine to process all the tables
Public Sub procTab()
Dim loopCnt, dropCnt, daysAhead As Integer
Dim sql, tabName, tmpStr, partName As String
Dim partRec As clsPart
Dim dbCmd As OleDbCommand = Conn.CreateCommand
Dim dr As OleDbDataReader
Dim partDate, limDate As DateTime
Dim curDate As DateTime = Now
Dim dropPartList As New ArrayList


For loopCnt = 0 To (tabCollection.Count - 1)
tabName = UCase(tabCollection(loopCnt).Attributes("Name").Value)
daysAhead = tabCollection(loopCnt).Attributes("Period").Value
Console.WriteLine(vbNewLine & "Table " & tabName)
dropPartList.Clear()

chkKeyType(tabName)

If keyType <> InvalidKey Then
limDate = curDate.AddDays(daysAhead).Date
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Date Limit is " & Format(limDate, "dd-MMM-yyyy HH:mm"))

sql = pfxStr & tabName & "' and partition_name not like '" & tabName & "0' order by " _
& "partseq"
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Get partition list sql is " & vbNewLine & sql)
Try
dbCmd.CommandText = sql
dr = dbCmd.ExecuteReader
If dr.HasRows Then
While dr.Read
tmpStr = dr.Item("high_value")
partDate = getPartDate(tmpStr)
partName = dr.Item("partition_name")
partRec = New clsPart
partRec.Name = partName
partRec.LimDate = partDate
Trace.WriteLineIf(generalSwitch.TraceVerbose, partName & " " & Format(partDate, "dd-MMM-yyyy"))
If DateTime.Compare(partDate, limDate) < 0 Then
dropPartList.Add(partRec)
Trace.WriteLineIf(generalSwitch.TraceVerbose, partRec.Name & " " & Format(partRec.LimDate, "dd-MMM-yyyy") & " should be dropped")
Else
Trace.WriteLineIf(generalSwitch.TraceVerbose, partRec.Name & " " & Format(partRec.LimDate, "dd-MMM-yyyy") & " does not qualify")
End If
End While
End If

Finally
If Not IsNothing(dr) Then
dr.Close()
End If
End Try

' If there are partitions that qualify for dropping
If dropPartList.Count > 0 Then
Console.WriteLine(vbNewLine & "Date Limit is " & Format(limDate, "dd-MMM-yyyy HH:mm"))
For dropCnt = 0 To (dropPartList.Count - 1)
partRec = dropPartList(dropCnt)
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Dropping " & partRec.Name & " with " & Format(partRec.LimDate, "dd-MMM-yyyy"))
sql = "alter table " & tabName & " drop partition " & partRec.Name
dbCmd.CommandText = sql
dbCmd.ExecuteNonQuery()
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Drop partition sql is " & vbNewLine & sql)
Console.WriteLine("Dropped " & partRec.Name & " with " & Format(partRec.LimDate, "dd-MMM-yyyy"))
Next
Else
Console.WriteLine("No partitions qualify")
End If
Else
Console.WriteLine(tabName & " is an invalid table")
End If
Next
dbCmd.Dispose()

End Sub

Public Shared Sub Main(ByVal args() As String)
Dim dPart As New DropPartDate
Dim tmpDate As DateTime = Now()
Dim elapseTime As TimeSpan

Console.WriteLine(vbNewLine & "Create Partition (Date) - Version " & dPart.Ver & vbNewLine & lineStr)
Console.WriteLine(Format(Now(), "dd-MMM-yyyy HH:mm"))

Dim writer As TextWriterTraceListener = New TextWriterTraceListener(System.Console.Out)
System.Diagnostics.Debug.Listeners.Add(writer)

If generalSwitch.TraceVerbose Then
Console.WriteLine("Trace is verbose")
End If

Try
' Checks to see if a parameter is provided
If args.Length > 0 Then

' If parameter is ? then display the help
If args(0) = "?" Then
Console.WriteLine("Usage: " & vbNewLine & "crePartDate <configFileName>")
Exit Sub
Else
dPart.cfgFile = args(0)
End If
End If

dPart.readXML()
If dPart.cfgFound Then
dPart.getTabs()

' List table only if trace is verbose
If generalSwitch.TraceVerbose Then
dPart.listTabs()
End If

dPart.ConnectDB()
dPart.procTab()
End If

Catch ex As Exception
Console.WriteLine(vbNewLine & "Error " & vbNewLine & ex.Message)
Trace.WriteIf(generalSwitch.TraceVerbose, ex.StackTrace)
Finally
dPart.closeDB()
End Try

elapseTime = Now.Subtract(tmpDate)
Console.WriteLine(vbNewLine & "Elapse time " & elapseTime.Seconds & " seconds")
Console.WriteLine("End time " & Format(Now(), "dd-MMM-yyyy HH:mm"))
Console.WriteLine("End Program" & vbNewLine & lineStr)
End Sub

End Class

Class clsPart
Dim vName As String
Dim vLim As Date

Public Property Name() As String
Get
Return vName
End Get
Set(ByVal Value As String)
vName = Value
End Set
End Property

Public Property LimDate() As Date
Get
Return vLim
End Get
Set(ByVal Value As Date)
vLim = Value
End Set
End Property
End Class

As in creating partition, I also use an xml instead of the config file. This is to allow me to drop partition from multiple database instance by supporting each database with a separate xml file. Here is a sample xml file:
<config>
<connStr Val="Provider=ORAOLEDB.Oracle;PASSWORD=tiger;User ID=scott;Data Source=orcl" />
<tab Name="TESTPART" Period="-200" />
<tab Name="TESTPART2" Period="-300" />
<tab Name="MFGTRANS" Period="-250" />
<tab Name="PARTMASTER" Period="-200" />
</config>

Monday, May 26, 2008

Log Parser

After starting to use the new log feature in Framework 2.x, how do you analyze the log file. There is a free tool provided by Microsoft to do just that. That tool can be obtained from here. However, it can be quite challenging since it is a line command utility. Also in order for it to work, you need to make sure you use , as the delimiter so that it can treat your log file as a CSV file.

Luckily somebody else had taken an initiative to make it better by coming up with a GUI interface called Visual LogParser, unfortunately, I found this to be lacking. It did not allow some of the options.

You can read more about the features of LogParser from Microsoft, or from another blogger here who has also linked some other resources.

Sunday, May 25, 2008

Enhanced Tracing in Framework 2.0

Following up on the previous post on logging, seems that the same mechanism can be used on the tracing. In the case of application with multiple modules, you can create multiple TraceSource and associate each TraceSource to each module. That way, you can trace output from each module separately.

To activate the tracesource, you just have to retrieve the tracesource in your program as follows:

Dim ts as new TraceSource("MainForm")

"MainForm" is the name of the tracesource as seen in the config file:
  <system.diagnostics>
<sources>
<!-- This section defines the logging configuration for My.Application.Log -->
<source name="MainForm" switchValue="All">
<listeners>
<add name="LogToFile"/>
</listeners>
</source>
</sources>
<switches>
<add name="DefaultSwitch" value="All" />
<add name="ActSwitch" value="ActivityTracing" />
<add name="VerSwitch" value="Verbose" />
</switches>
<sharedListeners>
<!-- add name="FileLog"
type="Microsoft.VisualBasic.Logging.FileLogTraceListener, Microsoft.VisualBasic, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL"
initializeData="FileLogWriter"/ -->
<!-- Uncomment the below section and replace APPLICATION_NAME with the name of your application to write to the Application Event Log -->
<!--<add name="EventLog" type="System.Diagnostics.EventLogTraceListener" initializeData="APPLICATION_NAME"/> -->
<add name="LogToFile" type="Microsoft.VisualBasic.Logging.FileLogTraceListener, Microsoft.VisualBasic, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL"
initializeData="FileLogWriter"
logFileCreationSchedule="Daily"
traceOutputOptions="DateTime"
BaseFileName="MainWin"
location="Custom"
CustomLocation="c:\log\"
Delimiter ="|"
AutoFlush="True"
Append="True"
/>
</sharedListeners>
</system.diagnostics>


This is superior to the one available in Framework 1.1 because, instead of using a If statement to check the tracelevel prior to performing Trace.WriteLine, you use ts.TraceEvent(TraceEventType.Information, 1, "Started Trace").

In framework 1.1, I facilitated this by creating a subroutine:

'==================================================
' performs a trace write if the switch is > then the pLevel
' If flg = 0 then do a trace.write
' if flg = 1 then do a trace.writeline
' =================================================
Sub traceMsg(ByRef tSwitch As TraceSwitch, ByVal pLevel As TraceLevel, _
ByVal pMsg As String, Optional ByVal traceCat As String = "Debug", _
Optional ByVal flg As Integer = 0)

If tSwitch.Level >= pLevel Then
If flg = 0 Then
Trace.Write(traceCat, pMsg)
Else
Trace.WriteLine(traceCat, pMsg)
End If

End If
End Sub

Application Logging

Some new classes were introduced to facilitate application logging. Previously, we had to depend on 3rd party libraries like log4net. One of the classes that are most useful is the FileLogTraceListener which provided autoarchiving by date or by application.

I started exploring the information using information from the this article.

First lets start with the app.config file:

<system.diagnostics>
<sources>
<!-- This section defines the logging configuration for My.Application.Log -->
<source name="DefaultSource" switchName="MSwitch">
<listeners>
<add name="LogToFile" />
</listeners>
</source>
</sources>
<switches>
<add name="MSwitch" value="All" />
</switches>
<sharedListeners>
<add name="FileLog"
type="Microsoft.VisualBasic.Logging.FileLogTraceListener, Microsoft.VisualBasic, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL"
initializeData="FileLogWriter"/>
<!-- Uncomment the below section and replace APPLICATION_NAME with the name of your application to write to the Application Event Log -->
<!--<add name="EventLog" type="System.Diagnostics.EventLogTraceListener" initializeData="APPLICATION_NAME"/> -->
<add name="LogToFile" type="Microsoft.VisualBasic.Logging.FileLogTraceListener, Microsoft.VisualBasic, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL"
initializeData="FileLogWriter"
logFileCreationSchedule="Daily"
traceOutputOptions="DateTime"
BaseFileName="W_Test"
location="Custom"
CustomLocation="c:\log\"
Delimiter ="|"
AutoFlush="True"
Append="True"
/>
</sharedListeners>
</system.diagnostics>

To use be able to use My.Application.log class, you need a source called DefaultSource, you can then point it to a switch name. In the switch name (or place the value directly using SwitchValue, you have an option to identify the level of logging (taken from the help file) :

ActivityTracing Enumeration
Description
Off Does not allow any events through.
Critical Allows only Critical events through.
Error Allows Critical and Error events through.
Warning Allows Critical, Error, and Warning events through.
Information Allows Critical, Error, Warning, and Information events through.
Verbose Allows Critical, Error, Warning, Information, and Verbose events through.
ActivityTracing Allows the Stop, Start, Suspend, Transfer, and Resume events through.
All Allows all events through.


You can then add various listeners. If you are using FileLogTraceListener,
  • you can specify the BaseFileName. This will be the used as your log file name.
  • specify whether or not to append to the log.
  • autoflush
  • location (if specify Custom then specify CustomLocation to tell it where to put the file).
  • logFileCreationSchedule whether to create Daily, Weekly or None.
  • Delimiter (character to use as delimiter between the columns in the log file).
  • TraceOutputOption (additional items to automatically add to the log file). If you use DateTime, it will provide it give you the time in GMT timezone.
Based on the above configuration, my log file will have the name W_Test-2008-05-24.log and the content will look something like this:

DefaultSource|Start|4|Entering Button2_click|2008-05-24 03:37:54Z
DefaultSource|Stop|5|Leaving Button2_click|2008-05-24 03:37:54Z
DefaultSource|Information|0|Going thru Button2|2008-05-24 03:37:54Z

Saturday, May 24, 2008

Changing Color in DataGridView based on content

The following is how it can be achieved:

Private Sub dgView_RowPostPaint(ByVal sender As Object, ByVal e As System.Windows.Forms.DataGridViewRowPostPaintEventArgs) Handles dgView.RowPostPaint
Dim vValue As String = ""
Dim vCnt As Integer = dgView.Rows.Count
Dim cellCnt As Integer

If Me.Visible Then
If vCnt > 0 Then
vValue = dgView.Rows(e.RowIndex).Cells(3).Value
If IsNothing(vValue) = False Then
If LCase(vValue).Contains("err") Then
For cellCnt = 0 To (dgView.Rows(e.RowIndex).Cells.Count - 1)
dgView.Rows(e.RowIndex).Cells(cellCnt).Style.ForeColor = Color.Red
Next
End If
End If
End If

End If
End Sub



In this example, I am looking at the fourth column and if it contains the word "err", the words will be turned red.



An earlier routine using RowsAddEvent did not work when I converted from an unsorted data to sorted data.

Friday, May 23, 2008

Removing Close Button From Windows

This snippet was obtained from a reply found here.

Private Declare Auto Function GetSystemMenu Lib "user32.dll" (ByVal hWnd As IntPtr, _
ByVal bRevert As Int32 _
) As IntPtr

Private Declare Auto Function GetMenuItemCount Lib "user32.dll" ( _
ByVal hMenu As IntPtr _
) As Int32

Private Declare Auto Function DrawMenuBar Lib "user32.dll" ( _
ByVal hWnd As IntPtr _
) As Int32

Private Declare Auto Function RemoveMenu Lib "user32.dll" ( _
ByVal hMenu As IntPtr, _
ByVal nPosition As Int32, _
ByVal wFlags As Int32 _
) As Int32

Private Const MF_BYPOSITION As Int32 = &H400
Private Const MF_REMOVE As Int32 = &H1000

Private Sub RemoveCloseButton(ByVal frmForm As Form)
Dim hMenu As IntPtr, n As Int32
hMenu = GetSystemMenu(frmForm.Handle, 0)
If Not hMenu.Equals(IntPtr.Zero) Then
n = GetMenuItemCount(hMenu)
If n > 0 Then
RemoveMenu(hMenu, n - 1, MF_BYPOSITION Or MF_REMOVE)
RemoveMenu(hMenu, n - 2, MF_BYPOSITION Or MF_REMOVE)
DrawMenuBar(frmForm.Handle)
End If
End If
End Sub


The other code found here doesn't work. It grays out the X button but it is still functional.

Monday, May 19, 2008

Application Scheduling

Sometimes it is necessary to perform activities on regular intervals such as:
  1. Sending daily email alert.
  2. Compiling wip @ a specific time.
  3. Compiling yield or transactions either shiftly or daily.
This can be easily done by writing a console (non GUI) application and then scheduling the application to run at regular intervals. With Windows 2000, Windows XP, and Windows 2003, you can use the Windows scheduler.

The Windows scheduler was first introduced in Windows NT but it did not provide a nice interface for adding to and viewing the schedule (using AT command - similar to the command available in Unix). This has since been improved.

Nice thing about the Windows Scheduler is you can set it to run as different users. Typically, most will run under Admin account. However, note that when it runs, the screen will only display in the console.

One of the benefit, Windows Scheduler has over the Unix (cronjob) is that it has a built-in mechanism that will ensure only a copy of the schedule item will run at any time. While Unix does not do that. For example, you may schedule extractData and schedule it to run every 3 minutes. In Unix, if the extract takes more than 3 minutes, you may end up with multiple copies running at the same time. This will not happen if scheduled in Window scheduler.

There are also some failsafe such as automatically abort if the job takes more than a specified amount of time.

Finally when we schedule a job we typically expect it to run automatically. There are some additional precaution we should take:
  • have a log file output so that we can trace when there are problems. Easiest way is to redirect the output of the program (for example extractData >> c:\log\extractData.log)
  • Have the application generate a heartbeat file each time it completes successfully and then have another program monitor the heartbeat file modified time. If the heartbeat update time exceeds a threshold, just have the application send an alert.

Saturday, May 17, 2008

Automating Creation of Oracle Partition

A typical way of creating Oracle partition is to partition by a column representing a date field. However, it is possible to represent the date either by a date type or as Varchar2. When representing the date as Varchar2, we normally represent it as "yyyymmdd".

Let me share with you a vb.net console program I wrote to automate the creation of partition.


First lets create two tables:

CREATE TABLE TESTPART
(RECID NUMBER,
LOTID VARCHAR2(20),
OLD_LOTID VARCHAR2(20),
ACTIVITY VARCHAR2(40),
TXNTIME CHAR(18),
EXTDATE VARCHAR2(15),
FROMTIME VARCHAR2(18),
TOTIME VARCHAR2(18),
NOTES VARCHAR2(255),
BRIEFDESC VARCHAR2(255),
DETAILDESC VARCHAR2(255)
)
PARTITION BY RANGE (EXTDATE)
(PARTITION TESTPART0 VALUES LESS THAN ('20010101 080001'));


CREATE TABLE TESTPART2
( TIME_DATE DATE,
LOTID VARCHAR2(20),
ACTIVITY VARCHAR2(40),
STEPNAME VARCHAR2(60 BYTE),
REASONCODE VARCHAR2(20 BYTE),
REASONQTY NUMBER,
CATEGORY VARCHAR2(6 BYTE),
BONUSQTY NUMBER
)
PARTITION BY RANGE (TIME_DATE)
(PARTITION FW_WOP_DEF0 VALUES LESS THAN (TO_DATE('20010101', 'YYYYMMDD'));

Both tables are partitioned by date but in the first one the date is stored as varchar2 and in the second one it is stored as a date field.

In order to start the process, we need to know how to get information on the partition. This is available in USER_TAB_PARTITIONS view. Next you need to know how the know how the data is stored. The column High_value, stores high value of the partition key (meaning the max value the partition will store. The problem with this is that the column type is long() and cannot be analyzed directly within PL/SQL which is why I had to use an external program to analyze the value otherwise, I would have created a stored procedure to automate the process.

If the partition key is of date type, then it will be stored as (TO_DATE(' 2001-01-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')). For varchar2, it will be stored as an actual value. This will allow us to check the partition key type that we use to perform the logic whether or not to create the partition.

To facilitate the automation, I follow the following convention:
  • Have a reference partition which is the table name followed by '0' , so in the example above the table is testpart and the reference partition is testpart0.
  • Then when I create the partition, I use a sequential numbering, ie. the partition will be testpart_1, testpart_2 etc.
  • This is used so that when I automate the dropping of partition, I will not accidentally drop the reference partition.
A few notes on the code:
  • ReadXML will read and load the xml into an XMLDocument and point the xmlnode to the right node (<config>).
  • chkKeyType will identify from the high_value whether the type used in the partition key column (varchar2 or Date).
  • getDate will get the highest high_value of the partitions that exist for the table.
  • procTab will go through the list of table provided in the XML. It will also compare the target date to create the partition and decide whether it is necessary to create the partition. If necessary then it will take the sequence from the last partition increment 1 and then use it as the name to create the new partition.

Finally, here is the code:
Imports System.Xml
Imports System.Data.OleDb

' Create Partition where the partition key is a date field
' Author Strovek
' Date Jul 21, 2005
' Revised
'
' Rewrote the program to recognize the different date partition key.
' date field or string with the format "yyyymmdd hh24miss"
'
' =============================================================================

Public Class CrePartDate

Const DateKey As String = "DateKey"
Const StrKey As String = "StrKey"
Const InvalidKey As String = "InvKey"
Const pfxStr As String = "SELECT PARTITION_NAME, HIGH_VALUE, " _
& "TO_NUMBER(SUBSTR(PARTITION_NAME, LENGTH(TABLE_NAME)+2)) PARTSEQ FROM USER_TAB_PARTITIONS WHERE TABLE_NAME = '"
Const lineStr As String = "================================"

Public Shared generalSwitch As New TraceSwitch("TraceLevelSwitch", "Entire Application")
Dim cfgFile As String = "..\crePartDate.xml"
Dim Conn As New OleDb.OleDbConnection
Dim connStr As String
Dim cfg As New XmlDocument
Dim root As XmlNode
Dim tabCollection As XmlNodeList
Dim keyType As String
Dim vConfigFound As Boolean = False

' Returns the version of this assembly
Public Shared Function Ver() As String
With _
System.Diagnostics.FileVersionInfo.GetVersionInfo(System.Reflection.Assembly.GetExecutingAssembly.Location)
Return .FileMajorPart & "." & .FileMinorPart & "." & .FileBuildPart & "." & .FilePrivatePart
End With
End Function

Public Function cfgFound() As Boolean
Return vConfigFound
End Function

' Property to set the configuration filename
Public Property cfgFName() As String
Get
cfgFName = cfgFile
End Get
Set(ByVal Val As String)
cfgFile = Val
End Set
End Property

' Reads the xml file
Public Sub readXML()
Dim flg As Boolean = False
Dim xNode As XmlNode
cfg.Load(cfgFile)
xNode = cfg.FirstChild
While True
Trace.WriteLineIf(generalSwitch.TraceVerbose, "xnode name is " & xNode.Name & "*")
If xNode.Name <> "config" Then
xNode = xNode.NextSibling
Else
vConfigFound = True
Exit While
End If
If IsNothing(xNode) Then
Exit While
End If
End While
root = xNode
End Sub

' Displays the Connection String from the xml file
Public Sub getConnStr()
Dim xNode As XmlNode
'xNode = cfg.SelectSingleNode("connStr")
xNode = root.SelectSingleNode("connStr")
Trace.WriteLineIf(generalSwitch.TraceVerbose, xNode.Attributes("Val").Value())
End Sub

' Gets the collection of Tables to process
Public Sub getTabs()
tabCollection = root.SelectNodes("tab")
End Sub

' Display the list of tables to be processed
Public Sub listTabs()
Dim cnt As Integer

For cnt = 0 To (tabCollection.Count - 1)
Console.Write(tabCollection(cnt).Attributes("Name").Value & " ")
Console.WriteLine(tabCollection(cnt).Attributes("Period").Value)
Next

End Sub

' Connect to database
Public Sub ConnectDB()
Dim connStr As String = root.SelectSingleNode("connStr").Attributes("Val").Value
Conn.ConnectionString = connStr
Conn.Open()
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Database connected")
End Sub

' Closes the database connection if it is Open
Public Sub closeDB()
If Conn.State = ConnectionState.Open Then
Conn.Close()
Trace.WriteLineIf(generalSwitch.TraceVerbose, "Database connection Closed")
End If
End Sub

' Verify the key type for the table
Sub chkKeyType(ByVal tabName As String)
Dim dbCmd As OleDbCommand = Conn.CreateCommand
Dim dr As OleDbDataReader
Dim sql, tmpStr, tmpStr2 As String
Dim tmpDate As DateTime
keyType = InvalidKey

sql = pfxStr & tabName & "' and partition_name = '" & tabName & "0'"
dbCmd.CommandText = sql
Try
dr = dbCmd.ExecuteReader
If dr.HasRows Then
dr.Read()
tmpStr = dr.Item("high_value")
If Left(tmpStr, 7) = "TO_DATE" Then
keyType = DateKey
Else
Try
tmpStr2 = tmpStr.Substring(1, 8)
tmpDate = Date.ParseExact(tmpStr2, "yyyyMMdd", Nothing)
keyType = StrKey
Catch ex As Exception
keyType = InvalidKey
End Try
End If
End If
Finally
If Not IsNothing(dr) Then
dr.Close()
End If
End Try

dbCmd.Dispose()
End Sub

' Extract the date from the highvalue field and return it as datetime
Function getDate(ByVal highValue As String) As DateTime
Dim tmpStr As String
Dim tmpDate As DateTime

tmpStr = highValue.Substring(highValue.IndexOf("'") + 2, 19)
Trace.WriteLineIf(generalSwitch.TraceVerbose, tmpStr)
tmpDate = DateTime.ParseExact(tmpStr, "yyyy-MM-dd HH:mm:ss", Nothing)

Return tmpDate.Date
End Function


' Decide which method to extract the date portion of the date from the highvalue column
Function getPartDate(ByVal highval As String) As DateTime
Dim tmpDate As DateTime
Dim tmpStr As String

If keyType = DateKey Then
tmpDate = getDate(highval)
Else
tmpStr = highval.Substring(1, 8)
tmpDate = DateTime.ParseExact(tmpStr, "yyyyMMdd", Nothing)
End If
Return tmpDate.Date
End Function

' Routine to process all the tables
Public Sub procTab()
Dim loopCnt, daysAhead, partseq As Integer
Dim tabName, tmpStr, partName, sql As String
Dim dbCmd As OleDbCommand = Conn.CreateCommand
Dim dr As OleDbDataReader
Dim partDate, limDate As DateTime
Dim curDate As DateTime = Now()
Dim crePartFlg As Boolean

For loopCnt = 0 To (tabCollection.Count - 1)
tabName = UCase(tabCollection(loopCnt).Attributes("Name").Value)
daysAhead = tabCollection(loopCnt).Attributes("Period").Value
Console.WriteLine(vbNewLine & "Table " & tabName & " ")
chkKeyType(tabName)
If keyType <> InvalidKey Then
crePartFlg = False
If daysAhead > 0 Then

limDate = curDate.AddDays(daysAhead).Date

sql = pfxStr & tabName & "' and partition_name not like '" & tabName & "0' order by " _
& "to_number(substr(partition_name, length(table_name)+2)) desc"
dbCmd.CommandText = sql
Try
dr = dbCmd.ExecuteReader
If dr.HasRows Then
dr.Read()
tmpStr = dr.Item("high_value")
partDate = getPartDate(tmpStr)
Console.WriteLine("Date limit is " & Format(limDate, "dd-MMM-yyyy HH:mm"))
Console.WriteLine("Part Limit is " & Format(partDate, "dd-MMM-yyyy HH:mm"))

If DateTime.Compare(partDate, limDate) < crepartflg =" True" partseq =" dr.Item(" crepartflg =" True" partseq =" 1" partname =" tabName" keytype =" DateKey" sql = "alter table " sql = "alter table " tmpstr = "Create partition script " commandtext =" sql" datetime =" Now()" textwritertracelistener =" New"> 0 Then

' If parameter is ? then display the help
If args(0) = "?" Then
Console.WriteLine("Usage: " & vbNewLine & "crePartDate
")
Exit Sub
Else
cPart.cfgFile = args(0)
End If
End If

cPart.readXML()
If cPart.cfgFound Then
cPart.getTabs()

' List table only if trace is verbose
If generalSwitch.TraceVerbose Then
cPart.listTabs()
End If

cPart.ConnectDB()
cPart.procTab()
End If

Catch ex As Exception
Console.WriteLine(vbNewLine & "Error " & vbNewLine & ex.Message)
Trace.WriteIf(generalSwitch.TraceVerbose, ex.StackTrace)
Finally
cPart.closeDB()
End Try

elapseTime = Now.Subtract(tmpDate)
Console.WriteLine(vbNewLine & "Elapse time " & elapseTime.Seconds & " seconds")
Console.WriteLine("End time " & Format(Now(), "dd-MMM-yyyy HH:mm"))
Console.WriteLine("End Program" & vbNewLine & lineStr)
End Sub

End Class

Since we typically may have multiple database instances or schema, I use an xml instead of the config file so that I can use one xml for each account/database and then have multiple tables in the xml. For example:
<?xml version="1.0" encoding="utf-8" ?>
<!--
Document : crePartDate.xml
Created on : Jul 21, 2005
Modified : Jul 25, 2005
Author : Strovek
Description:
Configuration file for crePart Project
-->
<config>
<connStr Val="Provider=ORAOLEDB.Oracle;PASSWORD=tiger;User ID=scott;Data Source=orcl" />
<tab Name="TESTPART" Period="7" />
<tab Name="TESTPART2" Period="7" />
</config>
<!--
Modification History
-->

The period means how many days ahead I need to create the partition, in the example ahead it is 7 days so if today is May 17, 2008, the program will create a partition with the high_value of May 24, 2008. So to run the program, just execute crePartDate crePartDate.xml. With this you can then use the Windows scheduler to run either daily, weekly etc.

This concept can be done in any other language like C# and java.

Wednesday, May 14, 2008

Interface Matters

One of the key things in programming is the interface. This is how the user interacts with our application. Take example of some of the programs we used to use. In word processing, we had WordStar, then WordPerfect and now Microsoft Word.

Why did Wordstar die off, it is due to the interface? WordPerfect was more intuitive than Wordstar. WordPerfect provided the first WYSIWYG. Then MS Word came along and provide an even more intuitive interface.

Another example is Lotus 1-2-3 again killed off by an application providing a more intuitive interface - namely Excel.

Word of caution though, intuition is very subjective and familiarity comes into play also. For example in some applications like web sites, there are certain expectations like the menu is on the top or side. So we have also got to take into consideration user expectation. Another example is when we drive a car, we expect a steering wheel and not a joystick.

Two years ago, I attended Lotusphere and one of the session covered was "Interface Matters". Go to the website. Even though they use Lotus Domino as a platform, some of the ideas covered can be used in any other platform.

Think of interface as packaging for your software. You can see some fantastic ideas out there such as Ipod, Macintosh. Elegance and simplicity is the key to a successful interface. Interface designing is an art as well as a science so have fun with it but make sure to get feedback from your users!!

Saturday, May 10, 2008

How to WaterMark an Image file

Nowadays, we place a lot of images on Internet which can be easily copied and used by others. One way is to place a watermark on the image so that even if others copy your images, you copyright or whatever note you have will still be in the image.

This is how you can do that using vb.net


Public Sub addWaterMark(ByVal frmFileName As String, ByVal toFileName As String, ByVal pStr As String)
Dim Img As Image
Dim g As Graphics
Dim brsh As Brush

Try
Img = Image.FromFile(frmFileName)
g = Graphics.FromImage(Img)
Dim fontFamily As New FontFamily("Arial")
Dim font As New Font( _
fontFamily, _
60, _
FontStyle.Regular, _
GraphicsUnit.Pixel)
brsh = Brushes.Gray
g.DrawString(pStr, font, Brushes.Gray, 300, 50)
g.DrawImage(Img, 0.0F, 0.0F)
Img.Save(toFileName)
Console.WriteLine("Watermark done")

Catch ex As Exception
Throw New Exception("[addWaterMark01]" & ex.Message)
End Try
End Sub

As shown above,
  1. First load the image file into an image object
  2. Create a Graphics object from the image object.
  3. Then write the message you want, using the font, color and size needed.
  4. Draw the Graphics back into the image.
  5. Then save the image to the new file (so you still have your original file.
If you want you can change it further to process a folder instead of a file so that you can perform the watermarking in batch mode prior to uploading into internet.

Thursday, May 8, 2008

Working in XML in VB.net

The easiest way to create xml from scratch is to use the XMLDocument class.

All XML needs to have a root node. To do that, just declare a new XMLDocument class object and then use it to create the root node.

For example if you want to create the following xml:

<Book>
<Title>Programmer's Journal</Title>
<Author Name="Me" />
</Book>


You can write something like this:

Imports System.Xml

Public Class clsTest

Private Shared Function genXML() As XmlNode
Dim xRoot, xNode As XmlNode
Dim xAttr As XmlAttribute
Dim xDoc As New XmlDocument

xRoot = xDoc.CreateElement("Book")

Try
xNode = xDoc.CreateElement("Title")
xNode.InnerText = "Programmer's Journal"
xRoot.AppendChild(xNode)
xNode = xDoc.CreateElement("Author")
xAttr = xDoc.CreateNode(XmlNodeType.Attribute, "Name", xNode.NamespaceURI)
xAttr.Value = "Me"
xNode.Attributes.Append(xAttr)
xRoot.AppendChild(xNode)

Catch ex As Exception
Throw New Exception("[genXML01]" & ex.Message)
End Try

Return xRoot
End Function

Public Shared Sub main(ByVal args() As String)
Console.WriteLine(clsTest.genXML.OuterXml)
End Sub

End Class

Running the program above will generate an output like this:

<Book><Title>Programmer's Journal</Title><Author Name="Me" /></Book>


Note that you use XMLDocument.CreateElement to create each node and then append subsequent nodes to root node. The values within the tags are call innerText and then you can also get the whole xml by getting the OuterXML property from the root node. Values within the nodes are called attributes and is generated using XMLDocument.CreateNode and specifying the node type.

One gotcha, you can only append XMLNodes that are created from the same XMLDocument as child to the nodes.

Wednesday, May 7, 2008

.Net IDE

When I started work with .Net several years back, I started out with Web Matrix. This only supports Framework 1.1 but it was sufficient to get me working, however, it lacks the intellisense. Then I found other free IDEs that help with my work such as SharpDeveloper. This was better and it provide a limited intellisense capability. SharpDeveloper had its own project files so was in a way similar to Visual Studio. However, the problem I had with it was if I was to add reference to an assembly it memorizes it. So if I was to refer to my custom assembly and then decide to recompile that assembly, it was necessary to go and remove it from the reference.

Snippet Compiler was also a very nice IDE which again provide some limited intellisense. However, when Framework 2.0 was released, majority of the IDEs broke.

For a time, I could not find any good IDE but finally Microsoft came out with the express edition for Visual Studio 2005. Currently there is also a Express edition for Visual Studio 2008. The express edition of Visual Studio provided most of the functionalities of Visual Studio with some limitations.

Main limitations are:
  • Instead of everything in one, you have different packages
    • Web design
    • Windows/Console app (1 for VB and 1 for C# and 1 for C++)
  • WorkFlow capability is removed from VS 2008 express.
However, if you are just looking for a good IDE to help with writing the code then I think it is more then sufficient. I love using the ability to open a folder as a website and then just writing the code. This is useful when I want to do some quick programming without bothering with projects and solutions. You can then use either command prompt to compile or use a tool called NANT to compile your codes.

Thursday, May 1, 2008

DataTable as a Database

Continuing the previous article on offline scallable computing, the we can use datatable as our database.

There are a many methods and classes that provide support this. First we have the DataSet which can be thought as a database containing many DataTables. You can save the dataset directly into an XML file or create a DataSet directly from the XML file.
  • Writing can be done using either DataSet.WriteXml(filePath, XmlWriteMode.WriteSchema).
  • Retrieving the data is back to the DataSet is done using DataSet.ReadXML(filePath).
You can then use DataView to facilitate the selection of data. For example if you have an employee table and you want to select an employee record using the dataview, you could do the following:

DIM dv as DataView

dv = New DataView(ds.Tables("EmpMaster"), "empno='111000'")


You can update the record retrieved in the DataView by performing a BeginEdit and EndEdit.

For example you can update the persons address as follows:

dv.AllowEdit = True 'This is because by default the view is read-only
dv.BeginEdit()
dv(0)("Address") = "111 AprilFool Lane"
dv.EndEdit()

This all happens in memory so after updating, you need to write the dataset back to the file. This is the only weakness of this approach since we will need to write back the whole dataset even though we just updated one column in one row.

Sunday, April 27, 2008

Portable Workspace

I am used to certain type of tools for my programming such as Visual Studio, Netbeans, Context etc. However, it is sometimes quite cumbersome to carry the desktop, notebook every where. With the new breed of portable storage devices like thumb drive and Free Agent, it is possible to carry your programming environment with you.

Ceedo comes with Free Agent. However, the type of applications you can install on it is limited - for example, it is not possible to install Visual Studio on it. A much better option is to use MojoPac. Of course certain software like Antivirus, or softwares that tie directly to the hardware still cannot be installed but it is still a much better option than Ceedo (comes with FreeAgent) or U3 (comes with selected ThumbDrives). Problem with MojoPac, is that it only works with Windows XP. Now that Microsoft is going to stop selling XP, this may not be an option anymore.

Other options like VMWare is more troublesome, you are required to purchase additional Windows License to operate.

However, it raises a problem, what happens if you lose or somebody steals your portable hard disk? The solution is to encrypt the data. One is to use the encryption that comes with Free Agent but it is all or nothing. There are alternatives. Two such alternatives are
The benefit is using either one of the above, is you create a file that contains your virtual volume. That makes it easy for you to specify the size of the volume and to back it up - just copy the file. FreeOTFE also comes with a PDA version, so you can create a virtual secured volume on your PDA to store your accounts/password.

Both software uses very high encryption. Check it out.

Saturday, April 26, 2008

Woes of a Programmer

Recently, we completed a Windows Application and started running it. This is application was written using Framework 3.5. Strange thing is it runs on several machines but not on two of them - we tried on 8 machines total. Of this 5 of them were identical in every way - Windows, patches etc. Whenever, we run this, it will crash - but only the portion that enables multi threading.

This reminds me of a similar problem I had several years back when I tried using Harvard Graphics on a Acer desktop where the graphics was really bad but not on other machines.

Another incident I experienced more than 15 years ago when we were using MS DOS. I wrote a Clipper program for my office. It copied into two identical machines. Again, it ran on one but not on the other.

I guess I was lucky to encounter all these. We can't blame the programming or the programmer. I wonder whether there are some minor bugs in the processor. Hmmm! This is probably the worst thing that can happen to a programmer. You write your program and tested everything but it does not work on the machine it was meant for.

Tracing in ASP.Net

Tracing in ASP.Net

Tracing can be enabled either at the page level or for the whole application.

Page level
To activate tracing for a specific page, just add trace=enabled in the page directive as in:

<%@ Page Language="VB" AutoEventWireup="false" CodeFile="Maint.aspx.vb" Inherits="Maint" Trace="True" %>

When enabled at the page level, the trace information will be appended at the bottom of the page.

Application level
The switch to activate it at the application level is in the web.config. The directive is as follows:

<trace enabled="true" requestLimit="10" pageOutput="false" traceMode="SortByTime" localOnly="false"/>

The above tag is within the system.web tag as in:
<system.web>
<trace enabled="true" requestLimit="10" pageOutput="false" traceMode="SortByTime" localOnly="false"/>
</system.web>


With the application level tag, you have additional flexibility:

pageOutput - true will produce the output to the bottom of each page within the application. false will not.

In addition, another output is available - trace.axd page will be available once the trace is enabled at the application level. A sample of trace.axd is shown here:



TraceSwitch
TraceSwitch can be used to control the level of tracing to be used in the application. There is two steps to this process:

1. Add an entry in the web.config.
2. Create an object to represent the switch.

The web.config entry looks like this:

<system.diagnostics>
<switches>
<add name="debugSwitch" value="4"/>
<!--
Value options are:
Off 0
Error 1
Warning 2
Info 3
Verbose 4
-->
</switches>
</system.diagnostics>


The above entry must be between the <configuration> and </configuration>. The comment is not necessary but is there to remind me what the values mean.

This is how you access the setting:

Dim dSwitch As New TraceSwitch("debugSwitch", "For TraceDebugging")

With this you have properties to identify the trace level:

.TraceError
.TraceWarning
.TraceInfo
.TraceVerbose


Note that the tracelevel is a hierarchy, which means the higher level will include all messages provided for the lower level. So the value will be <=. E.g. TraceInfo will include messages for TraceInfo, TraceWarning and TraceError.

To help reduce the amount of code, I created a helper subroutine to output the trace messages:

Sub traceMsg(ByVal pLevel As TraceLevel, ByVal pMsg As String)
If dSwitch.Level >= pLevel Then
Trace.Write("Debug", pMsg)
End If
End Sub


Trace.Write is used to output the message to the trace listener - in this case the page trace.axd. Note that the tracelevel is in a hierarchy. It means TraceWarning will show messages for TraceWarning and TraceError - TraceInfo will include TraceWarning and TraceError.

To facilitate coding in multiple applications, you can place this in a common library as a module or a share subroutine and compile it into a library (.dll) and then refer to the library in your subsequent projects.

The following is a sample on how to use the tracing.

Web.config

<?xml version="1.0"?>
<configuration>
<system.web>
<trace enabled="true" requestLimit="10" pageOutput="false" traceMode="SortByTime" localOnly="false"/>
<system.diagnostics>
<switches>
<add name="debugSwitch" value="4"/>
<!--
Value options are:
Off 0
Error 1
Warning 2
Info 3
Verbose 4
-->
</switches>
</system.diagnostics>
</configuration>


Again the key section here is the system.diagnostics and trace.

Friday, April 25, 2008

Writing .Net application without Visual Studio

Even though it is much easier to write .Net application using Visual Studio but it is still possible to write applications using just a notepad. If you have Windows XP, Framework 1.1 comes pre-installed. There is currently two main Frameworks available 1.1 and 2.0. Framework 3.0 and Framework 3.5 are just Framework 2.0 with additional add-ons.

If you have Framework 1.1 installed, the folder will look like this:
C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322

While if you have Framework 2.0 with 3.x installed, you will see
C:\WINDOWS\Microsoft.NET\Framework\v2.0 and then see folder below it.

To compile a program, you need you just need to specify the proper compiler to compile it. Suppose you wrote a program called it HelloWorld.vb, you compile it using the the following command (then it will generate the application HelloWorld.exe):

C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\vbc HelloWorld.vb

If your program refers to other libraries then you add /r: followed by the list of assemblies it refers to. For example, /r:System.data.dll,System.Web.xml.

For Asp.net applications, you will need to compile it .dll files, so add /t:library. You can also use /o:HelloNewWorld.exe to change the output filename. To compile csc intead of vbc to compile CSharp programs.