[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

SV: SV: Converting from UTF16LE to Ascii



Hi Scott,
 
I took your advice and just read the IFS file into memory, convert it and write it back out.
 
I then used this command to get it to a pf:
CPYFRMIMPF FROMSTMF('/magnek/mms2.txt') TOFILE(MAGNEK/MMS) RCDDLM(*CRLF) FLDDLM(*TAB) DECPNT(*PERIOD) RPLNULLVAL(*FLDDFT)                                 
 
Here is the test program (everything is copied from statements that Scott has done before):
 
H DFTACTGRP(*NO) ACTGRP(*new) bnddir('QC2LE')                     
                                                                  
 /copy ifsio_h                                                   
 /copy iconv_h                                                    
                                                                  
D ReportError     PR                                              
D   peMsg                      256a   varying const               
D                                     options(*nopass)            
                                                                  
D CCSID_UTF16     c                   1200                        
D fd1             s             10i 0 inz(-1)                     
D fd2             s             10i 0 inz(-1)                     
D info            ds                  likeds(statds)              
D len             s             10i 0                             
D F1_size         s             10I 0                             
D memblock1       s          16383c   based(p_memblock1)
D memblock2       s          16383c   based(p_memblock2) ccsid(1200)  
D inpptr          s               *                                   
D outptr          s               *
D inpleft         s             10u 0                                  
D outleft         s             10u 0                                  
D outneed         s             10u 0                                  
D outsize         s             10u 0                                  
D ifs1            s             96a                                    
D ifs2            s             96a                                    
                                                                       
 /free                                                                 
   *inlr = *on;                                                        
   ifs1 = '/magnek/mms.txt';    //UTF-16le unpacked from zip to IFS    
   ifs2 = '/magnek/mms2.txt';   //UTF-16 (CCSID(1200) manually created.
                                                                       
   // --------------------------------------------                     
   //  Open the UTF-16le file:  Read only                              
   // --------------------------------------------                     
                                                                       
   fd1 = open( %trimr(ifs1) : O_RDONLY );                              
   if (fd1 = -1);                                                      
      ReportError();                                                   
      return;
   endif;                                             

   //---------------------------------------------
   //Open the UTF-16 file (CCSID(1200)
   //---------------------------------------------
                                                   
   fd2 = open( %trimr(ifs2)                           
            : O_RDWR + O_LARGEFILE                    
               + O_TEXTDATA + O_CCSID                 
               + O_SHARE_NONE                         
            : 0                                       
            : CCSID_UTF16);                           
   if (fd2 = -1);                                     
      ReportError();                                  
      return;                                         
   endif;   
                                          
   // --------------------------------------------    
   //  Figure out how big the file is (in bytes)      
   //  and reserve a block of memory to load it       
   //  into.                                          
   // --------------------------------------------    
                                                      
   if fstat(fd1: info) = -1;                          
      callp close(fd1);                               
      ReportError();                                 
      return;                                        
   endif;                                            
   F1_size = info.st_size;                           
                                                     
   p_memblock1 = %alloc(F1_size);                    
   p_memblock2 = %alloc(F1_size);                    
                                                     
   // --------------------------------------------   
   //  Read IFS1 into the block of                   
   //  memory.                                       
   // --------------------------------------------   
                                                     
   lseek(fd1: 0: SEEK_SET);                          
   len = read(fd1: p_memblock1: F1_size); 
           
   // --------------------------------------------   
   //  Convert from UTF-16le to UTF-16 (CCSID 1200)  
   //  memory.                                       
   // --------------------------------------------   
   inpptr = p_memblock1;                             
   inpleft = len;                              
   outleft = len;                              
   outptr  = p_memblock2;                      
   if QlgTransformUCSData( 050042              
                        : inpptr               
                        : inpleft              
                        : outptr               
                        : outleft              
                        : outneed ) <> 0;      
      ReportError();                           
      return;                                  
   endif;                                      
                                               
   // Write the result to IFS2 file            
   callp write(fd2: p_memblock2: len);
         
   callp close(fd1);                           
   callp close(fd2);                           
                                               
   dealloc p_memblock1;                        
   dealloc p_memblock2;                        
   return;                                                           
                                                                     
 /end-free                                                           
                                                                     
                                                                     
 *+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++  
 * ReportError():  Send an escape message explaining any errors      
 *                 that occurred.                                    
 *                                                                   
 *  This function requires binding directory QC2LE in order          
 *  to access the __errno() function.                                
 *+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++  
P ReportError     B                                                  
D ReportError     PI                                                 
D   peMsg                      256a   varying const                  
D                                     options(*nopass)               
                                                                     
D get_errno       PR              *   ExtProc('__errno')             
D errno           s             10I 0 based(p_errno)                 
D QMHSNDPM        PR                  ExtPgm('QMHSNDPM')       
D   MessageID                    7A   Const                    
D   QualMsgF                    20A   Const                    
D   MsgData                  32767A   Const options(*varsize)  
D   MsgDtaLen                   10I 0 Const                    
D   MsgType                     10A   Const                    
D   CallStkEnt                  10A   Const                    
D   CallStkCnt                  10I 0 Const                    
D   MessageKey                   4A                            
D   ErrorCode                 8192A   options(*varsize)        
                                                               
D ErrorCode       DS                  qualified                
D  BytesProv                    10I 0 inz(0)                   
D  BytesAvail                   10I 0 inz(0)                   
                                                               
D MsgKey          S              4A                            
D MsgID           s              7A                            
D Msg             s            256a   varying inz('')          
                                                               
 /free                                                         
                                                        
    if %parms >= 1;                                     
       MsgID = 'CPF9897';                               
       Msg   = peMsg;                                   
    else;                                               
       p_errno = get_errno();                           
       MsgID = 'CPE' + %editc(%dec(errno:4:0):'X');     
    endif;                                              
                                                        
    QMHSNDPM( MsgID                                     
            : 'QCPFMSG   *LIBL'                         
            : Msg                                       
            : %len(Msg)                                 
            : '*ESCAPE'                                 
            : '*PGMBDY'                                 
            : 1                                         
            : MsgKey                                    
            : ErrorCode         );                      
                                                        
 /end-free   
P                 E 
                                           
 
                                                            
 
                                 
 
          
 
 

________________________________

Fra: ftpapi-bounces@xxxxxxxxxxxxxxxxxxxxxx på vegne av Magne Kofoed
Sendt: ti 18.10.2011 08:39
Til: HTTPAPI and FTPAPI Projects
Emne: SV: SV: Converting from UTF16LE to Ascii


Hi Scott,
 
this is a tab separated file, not xml.
 
Best regards,
Magne
 

________________________________

Fra: ftpapi-bounces@xxxxxxxxxxxxxxxxxxxxxx på vegne av Scott Klement
Sendt: ti 18.10.2011 04:58
Til: ftpapi@xxxxxxxxxxxxxxxxxxxxxx
Emne: Re: SV: Converting from UTF16LE to Ascii



Hi Magne,

Frankly, for what you're doing, I wouldn't write the code this way at
all.  That code was intended to read one line at a time, looking for
CRLF type delimiters.  It makes sense for a CSV file.

For an XML file, I don't think I'd do that, I'd just read the file into
memory, convert it, and write it back out.  You don't care if it's
organized into lines...

On 10/17/2011 3:02 PM, Magne Kofoed wrote:
> Hi,
>
> I changed one stmt (the data variable) in the readlineUTF procedure:
>      if pos = 0;
>         data += inpbuf.buf;
>         inpbuf.pos += %len(inpbuf.buf);
>      else;
>         len = (pos - inpbuf.pos) + %len(EOL);
>         // data  += %subst(inpbuf.buf:inpbuf.pos:len);
>         data  = %subst(inpbuf.buf:inpbuf.pos:len);
>         inpbuf.pos += len;
>      endif;
>
> Works better now, but there is still one more bug.
> Seems like some data is missing in the beginning of the last data record in every buffer.
> Or the first record after a new read into the buffer that is missing the data.
> Have to debug more...
>
> Mvh
> Magne
>
>
> ________________________________
>
> Fra: ftpapi-bounces@xxxxxxxxxxxxxxxxxxxxxx på vegne av Magne Kofoed
> Sendt: ma 17.10.2011 18:49
> Til: HTTPAPI and FTPAPI Projects
> Emne: SV: Converting from UTF16LE to Ascii
>
>
>   Hi Jan,
>
> thanks, yes, I found the example program and use it to write to a pf.
> Scott, thank you for this example program.
>
> But I beleave there is a bug here, sometimes the variable UTFLINE has the whole buffer when the program writes to the pf. I'll try to debug and return to the list if I find out what's wrong.
>
> Best regards,
> Magne
>
>
> ________________________________
>
> Fra: ftpapi-bounces@xxxxxxxxxxxxxxxxxxxxxx på vegne av Jan Grove Vejlstrup
> Sendt: ma 17.10.2011 14:55
> Til: HTTPAPI and FTPAPI Projects
> Emne: Re: Converting from UTF16LE to Ascii
>
>
>
> Hi Magne
>
> Scott Klement wrote a little conversion-program for this.
>
> You can find it here:
>
> http://forums.systeminetwork.com/isnetforums/showthread.php?t=145571
>
>
> Best regards
>
> Jan
> Am 14.10.2011 19:06, schrieb Scott Klement:
>> Hi Magne,
>>
>> The file you have is UTF-16LE, but CCSID 1200 is UTF-16.  The "LE"
>> stands for "little endian", whereas CCSID 1200 is "big endian."
>>
>> IBM does not officially support UTF-16LE (CCSID 1202) anywhere on the
>> system as far as I know -- the only IBM-approved way to deal with
>> UTF-16LE is by converting it from UTF-16LE to UTF-16 (CCSID 1200) with
>> the QlgTransformUCSData() API.
>>
>> Once converted to UTF-16 (which is fully supported), you can mark it as
>> CCSID 1200 and deal with it normally.
>>
>>
>> On 10/14/2011 3:57 AM, Magne Kofoed wrote:
>>>       Hi Scott,
>>>
>>>
>>>       I got a file in Unicode (UTF16LE) and tries to import to a pf.
>>>
>>>       I have read a couple of your articles about this and got a few leads
>>>       on how to do this, but could not find how to get from Unicode to
>>>       ascii.
>>>
>>>
>>>       1.      I store the file in IFS, it gets CCSID = 1252.  I can view the
>>>       file with wrklnk and option 5, but I guess it's still Unicode since
>>>       I've just stored it.
>>>
>>>       2.      I tried to use these commands with no success:
>>>
>>>       CHGATR OB(`/magnek/mms1.txt') ATR(*CCSID) VALUE(1200)
>>>
>>>       I can not see any characters with wrklnk and option 5. But wordpad
>>>       shows the file ok.
>>>
>>>       CHGATR OB(`/magnek/mms1.txt') ATR(*CCSID) VALUE(819)
>>>
>>>       Same result as in nbr 1.
>>>
>>>       3.      I tried to use CPY fromccsid(1200) toccsid(819). Same result
>>>       as in nbr 1.
>>>
>>>       4.      I tried to use CPYFRMSTMF and CPYFRMIMPF with no success.
>>>
>>>       How can I get this UTF16LE file from the IFS to a PF?
>>>
>>>
>>>       Best regards,
>>>
>>>       Magne
>>>
>> -----------------------------------------------------------------------
>> This is the FTPAPI mailing list.  To unsubscribe, please go to:
>> http://www.scottklement.com/mailman/listinfo/ftpapi
>> -----------------------------------------------------------------------
>>
>>
>
>
>
>
> -----------------------------------------------------------------------
> This is the FTPAPI mailing list.  To unsubscribe, please go to:
> http://www.scottklement.com/mailman/listinfo/ftpapi
> -----------------------------------------------------------------------
-----------------------------------------------------------------------
This is the FTPAPI mailing list.  To unsubscribe, please go to:
http://www.scottklement.com/mailman/listinfo/ftpapi
-----------------------------------------------------------------------


<<winmail.dat>>

-----------------------------------------------------------------------
This is the FTPAPI mailing list.  To unsubscribe, please go to:
http://www.scottklement.com/mailman/listinfo/ftpapi
-----------------------------------------------------------------------