uPortal IRC Logs-2011-02-02

Owned by Apereo Infrastructure
Last updated: Feb 02, 2011Version comment
8 min read

[10:07:52 CST(-0600)] * Topic is '##uportal will be moving to #jasig-uportal on 2/7/11' set by EricDalquist on 10:07:52 CST(-0600)
[10:09:14 CST(-0600)] <athena> hey EricDalquist - so it's starting to seem like escaping ampersand parameter separators in URLs might not be a good idea
[10:09:23 CST(-0600)] <EricDalquist> ok
[10:09:29 CST(-0600)] <EricDalquist> what is it breaking?
[10:09:40 CST(-0600)] <athena> tomcat's HttpServletRequest implementation
[10:09:47 CST(-0600)] <EricDalquist> hah
[10:10:06 CST(-0600)] <athena> we have a few places where we rely on getParameterMap, and that then spits out parameter names like "amp;name"
[10:10:15 CST(-0600)] <EricDalquist> uhg
[10:10:29 CST(-0600)] <athena> might be a config option or something, but honestly, i don't want to change something like that if it means it could break people's existing code
[10:10:34 CST(-0600)] <EricDalquist> can you verify that is coming from tomcat's request?
[10:10:38 CST(-0600)] <EricDalquist> we have a bunch of request wrappers
[10:10:48 CST(-0600)] <EricDalquist> and we may not be doing the parameter escaping correctly
[10:10:54 CST(-0600)] <athena> pretty sure it's tomcat
[10:11:01 CST(-0600)] <athena> even looked at their source
[10:11:08 CST(-0600)] <EricDalquist> ah ok
[10:11:39 CST(-0600)] <athena> http://www.docjar.com/html/api/org/apache/tomcat/util/http/Parameters.java.html
[10:12:11 CST(-0600)] <athena> around line 500 starts havign some code w/ = and & characters
[10:12:31 CST(-0600)] <athena> so seems like our options are to have transitional and just have all those show up as errors
[10:12:34 CST(-0600)] <athena> or rethink xhtml?
[10:12:40 CST(-0600)] <EricDalquist> hrm
[10:12:52 CST(-0600)] <EricDalquist> well I really don't think xhtml is an option
[10:12:56 CST(-0600)] <athena> dunno if html5 changes anything in this
[10:13:03 CST(-0600)] <athena> because it means all the portlets have to conform?
[10:13:06 CST(-0600)] <EricDalquist> since if the page isn't 100% valid XML you get nothing useful
[10:13:13 CST(-0600)] <athena> yeah
[10:13:14 CST(-0600)] <EricDalquist> the browser just displays an XML parsing error
[10:13:27 CST(-0600)] <EricDalquist> and that would essentially make web proxy useless
[10:13:46 CST(-0600)] <athena> dunno if it could be an option for places that really want it and control all their content
[10:14:12 CST(-0600)] <EricDalquist> it seems like the user's browser should be un-escaping those &amp;
[10:14:42 CST(-0600)] <athena> hm.
[10:15:02 CST(-0600)] <EricDalquist> also in trunk it might be feasible to let this be more user-selectible
[10:15:09 CST(-0600)] <EricDalquist> er deployer selectible
[10:15:29 CST(-0600)] <athena> well, whether you want HTML or XHTML is at least configurable in 3.2
[10:15:40 CST(-0600)] <EricDalquist> except for the content type issue
[10:15:44 CST(-0600)] <EricDalquist> but that may not be too hard to fix
[10:15:53 CST(-0600)] <athena> yeah
[10:15:56 CST(-0600)] <EricDalquist> I'd encourage you to try it and see what works
[10:15:59 CST(-0600)] <athena> ok
[10:16:28 CST(-0600)] <athena> from a marketing standpoint, it'd be nice to be able to say all our stuff validates
[10:16:39 CST(-0600)] <EricDalquist> yeah
[10:18:01 CST(-0600)] <EricDalquist> I know that the bookmarks portlet for sure doesn't output valid html
[10:19:25 CST(-0600)] <athena> oooh i wonder about this
[10:19:43 CST(-0600)] <athena> i think maybe actually it is working but the portlet parameters are getting double escaped
[10:19:47 CST(-0600)] <athena> interesting.
[10:19:58 CST(-0600)] <EricDalquist> that could be
[10:20:07 CST(-0600)] <EricDalquist> there are a lot of layers around parameter handling :/.
[10:21:12 CST(-0600)] <athena> yeah, sounds like it
[10:39:18 CST(-0600)] <athena> hmm, making progress
[10:39:31 CST(-0600)] <athena> that dynamic title test - which portlet is that?
[10:40:12 CST(-0600)] <EricDalquist> jasig test portlet
[10:41:24 CST(-0600)] <athena> hmmm
[10:41:45 CST(-0600)] <athena> so right now the other test portlet renders escaped urls, but that one doesn't
[10:41:59 CST(-0600)] <athena> any ideas what's different?
[10:43:07 CST(-0600)] <EricDalquist> huh
[10:43:11 CST(-0600)] <EricDalquist> not sure
[10:44:04 CST(-0600)] <athena> weird.
[10:44:13 CST(-0600)] <athena> i mean it looks like they both use the portlet jsp tags
[11:14:46 CST(-0600)] <athena> wonder if there's any chance they're using different encodings? duno
[12:12:54 CST(-0600)] <athena> ok EricDalquist, so as best i can tell most of our portlets don't output escaped ampersands
[12:12:58 CST(-0600)] <athena> but the pluto testsuite does
[12:13:05 CST(-0600)] <athena> wondering if it has to do w/ spring portlets or something
[14:30:45 CST(-0600)] <rickard> athena: You haven't by any chance had time to look at the UTF8 issues with the Simple Content Portlet?
[14:30:59 CST(-0600)] <rickard> Hi, by the way (smile)
[14:31:30 CST(-0600)] <athena> hey!
[14:31:46 CST(-0600)] <athena> i think the UTF-8 issues are actually coming from ckeditor
[14:31:56 CST(-0600)] <athena> i haven't had a lot of time, but i played around with it a bit and that's sort of what it looks like
[14:32:03 CST(-0600)] <athena> have you had a chance to look at their documentation at all?
[14:32:20 CST(-0600)] <athena> seems like there are some config options for better i18n character handling
[14:36:59 CST(-0600)] <EricDalquist> weird the pluto test suite encoding issues
[14:37:11 CST(-0600)] <EricDalquist> they may be rendering the URLs in a different manner
[14:37:18 CST(-0600)] <EricDalquist> I'd have to look at the source to figure out whats up
[14:37:27 CST(-0600)] <athena> it's weird - both portlets are using the portlet JSP tag
[14:37:49 CST(-0600)] <EricDalquist> are there any other JSP level directives the pluto portlets are using?
[14:38:34 CST(-0600)] <athena> don't see anything interesting at the top
[14:38:41 CST(-0600)] <athena> just the taglib declaration
[14:38:50 CST(-0600)] <athena> they're not setting content-type at the top
[15:11:14 CST(-0600)] * Topic is '##uportal will be moving to #jasig-uportal on 2/7/11' set by EricDalquist!~apollo@76.210.64.222 on 10:07:52 CST(-0600)
[15:20:22 CST(-0600)] <rickard> athena: Hmm, ok. I'll see if I can have a look at it tomorrow. Sounds unlikely that something so widespread as ckeditor would have issues with utf8 though.
[15:20:52 CST(-0600)] <athena> let me find the page, hang on
[15:21:43 CST(-0600)] <athena> http://cksource.com/forums/viewtopic.php?f=6&amp;t=15356
[15:22:01 CST(-0600)] <athena> it sounds like by default ckeditor doesn't use utf8
[15:22:44 CST(-0600)] <athena> http://docs.cksource.com/FCKeditor_2.x/Developers_Guide/Configuration/Configuration_Options/ProcessHTMLEntities
[15:22:58 CST(-0600)] <rickard> Yeah, but that's just HTML entities.
[15:23:17 CST(-0600)] <rickard> Posting more exotic characters (such as japanese) breaks it completely.
[15:24:04 CST(-0600)] <athena> totally
[15:24:11 CST(-0600)] <athena> i tried that and the results were kinda tragic
[15:24:56 CST(-0600)] <rickard> I was concinved I hade found the problem when I notice the MySQL tables created by uPortal were Latin1, but changing them to utf8 didn't help.
[15:25:02 CST(-0600)] <rickard> noticed
[15:25:18 CST(-0600)] <athena> yeah i don't think it's database related - looks like the content is messed up before it's ever persisted
[15:25:32 CST(-0600)] <rickard> Is that by design by the way? Shouldn't the tables be utf8?
[15:25:48 CST(-0600)] <athena> i think probably the tables are created in whatever the default for the database is?
[15:25:55 CST(-0600)] <rickard> You can put utf8 data into a latin1 database, but stuff like substring() etc will break.
[15:25:58 CST(-0600)] <athena> i'd probably try creating the database itself as utf-8
[15:26:23 CST(-0600)] <rickard> Yeah, but it wouldn't hurt for uPortal to force it to utf8 (if possible).
[15:26:47 CST(-0600)] <holdorph> uportal doesn't create the database
[15:26:53 CST(-0600)] <holdorph> it only creates the tables
[15:26:56 CST(-0600)] <athena> hibernate/jpa is doing all the table creation
[15:27:05 CST(-0600)] <athena> so uportal's not even really explicitly doing that
[15:27:11 CST(-0600)] <rickard> Ok.
[15:27:25 CST(-0600)] <holdorph> right, but it is indirectly by using hibernate and the end user specifying the dialect
[15:27:28 CST(-0600)] <athena> i know sakai explicitly recommends mysql users create their database w/ utf8 encoding
[15:27:28 CST(-0600)] <rickard> In other words, if the database default is utf-8, the tables will be utf-8 accordingly?
[15:27:42 CST(-0600)] <holdorph> but the point was it should be a database default and uportal doesn't do the create database, even through a library
[15:27:43 CST(-0600)] <athena> i believe so
[15:28:04 CST(-0600)] <rickard> Ok. Might be a good point to include in the wiki under the mysql specific stuff.
[15:28:11 CST(-0600)] <athena> good point
[15:29:26 CST(-0600)] <rickard> While trying to figure out what's wrong with the simple content portlet, I realized I needed a way to debug this. If not in "real time", at least by logging stuff. Is there a simple way to write the e.g. uPortal.log from within a portlet?
[15:29:34 CST(-0600)] <holdorph> is the 'create database' command in the wiki anywhere right now?
[15:29:44 CST(-0600)] <rickard> holdorph:I don't believe so.
[15:30:10 CST(-0600)] <holdorph> because it's really only part of that command.
[15:30:45 CST(-0600)] <rickard> https://wiki.jasig.org/display/UPM32/MySQL
[15:30:53 CST(-0600)] <rickard> There's the mysql stuff
[15:31:24 CST(-0600)] <holdorph> https://confluence.sakaiproject.org/display/DOC/MySQL+Sakai+database+%282.x%29
[15:31:29 CST(-0600)] <holdorph> is what sakai provides
[15:32:23 CST(-0600)] <rickard> "Create a Sakai database with a default character set of UTF-8"
[15:32:30 CST(-0600)] <rickard> Just that would suffice.
[15:32:43 CST(-0600)] <athena> lol
[15:33:14 CST(-0600)] * rickard hopes athena found a stupid bug somewhere in the portlet.
[15:33:50 CST(-0600)] <athena> i unfortunately can't look at it right now, but i'll take another look later to see if there's something going on beyond the ckeditor issues
[15:34:09 CST(-0600)] <athena> i know tomcat doesn't default to utf-8, so that may be part of the problem
[15:34:35 CST(-0600)] <rickard> Hmm, ok. I'll see if I can have another look tomorrow.
[15:35:43 CST(-0600)] <holdorph> again, from the sakai community, turning on UTF-8 in tomcat: https://confluence.sakaiproject.org/display/DOC/Tomcat+UTF-8+%282.x%29
[15:36:54 CST(-0600)] <athena> more generally: http://wiki.apache.org/tomcat/FAQ/CharacterEncoding
[15:37:11 CST(-0600)] <athena> i suspect that the part about post parameters will be relevant to this portlet
[15:38:30 CST(-0600)] <holdorph> yeah, but the only 'tomcat' change, is the one listed on the sakai page, which is shorter to read (wink)
[15:39:56 CST(-0600)] <rickard> I doubt the URIEncoding parameter has anything to do with this though.
[15:40:10 CST(-0600)] <rickard> That's just for GET parameters.
[15:40:34 CST(-0600)] <holdorph> probably not, but it's the only thing you need to do at a tomcat configuration file level, for utf-8
[15:40:39 CST(-0600)] <holdorph> everything else is inside the application
[15:41:27 CST(-0600)] <rickard> True
[15:42:02 CST(-0600)] <rickard> I'll see what I can come up with tomorrow. Thanks for the pointers. Cheers
[15:43:35 CST(-0600)] <athena> good luck!
[15:44:12 CST(-0600)] <EricDalquist> so as the topic says we'll be moving from ##uportal to #jasig-uportal next week
[15:44:39 CST(-0600)] <EricDalquist> at that point joining this channel should redirect you to #jasig-uportal automatically
[15:45:03 CST(-0600)] <athena> hurray!
[15:45:11 CST(-0600)] <EricDalquist> the logbot is already over there
[15:46:52 CST(-0600)] * holdorph looks around
[15:47:03 CST(-0600)] <EricDalquist> welcome holdorph
[15:47:20 CST(-0600)] <holdorph> pretty clean looking, but at the same time still feels like home
[15:47:36 CST(-0600)] <EricDalquist> lol
[16:17:06 CST(-0600)] <EricDalquist> athena lfuller and holdorph do you have your nicknames registered with Freenode?
[16:17:26 CST(-0600)] <athena> yep
[16:17:32 CST(-0600)] <holdorph> nope. never saw a need to register mine. it was always available (smile)
[16:17:39 CST(-0600)] <holdorph> is it something I should do?
[16:17:39 CST(-0600)] <lfuller> same here
[16:17:50 CST(-0600)] <lfuller> lfuller and lennard are not very popular(smile)
[16:17:56 CST(-0600)] <EricDalquist> well in the new channel I'm going to add auto-op privs for all the uportal developers
[16:18:07 CST(-0600)] <EricDalquist> so there is more than just me in the rare chance someone decides to harrass the channel
[16:18:14 CST(-0600)] <EricDalquist> but you have to have a registered nickname
[16:18:21 CST(-0600)] * holdorph goes to look to see how to register
[16:18:45 CST(-0600)] <lfuller> (smile)
[16:20:24 CST(-0600)] <EricDalquist> hrm I tried adding permissions to athena and it came back that it did it to athena7
[16:20:42 CST(-0600)] <athena> hm, think that should be my alternate nick
[16:20:52 CST(-0600)] <EricDalquist> can you try joining #jasig-uportal
[16:21:12 CST(-0600)] * athena waves
[16:21:17 CST(-0600)] <athena> i have a star!
[16:21:23 CST(-0600)] <EricDalquist> yay it worked
[16:21:41 CST(-0600)] <holdorph> ok, i think holdorph is registered to holdorph@gmail.com
[16:22:08 CST(-0600)] <EricDalquist> worked
[16:22:09 CST(-0600)] <EricDalquist> great
[16:21:43 CST(-0600)] <athena> interesting - looks like maybe the whole account is registered first as athena7
[16:22:16 CST(-0600)] <EricDalquist> yeah
[16:22:39 CST(-0600)] <EricDalquist> so you should get auto-opd by chanserv when you join
[16:22:46 CST(-0600)] <holdorph> cool
[16:23:05 CST(-0600)] <EricDalquist> as other committers are in here we can do the same for them
[16:23:10 CST(-0600)] <EricDalquist> actually any of you can
[16:23:35 CST(-0600)] <EricDalquist> by doing "/msg chanserv flags #jasig-uportal UsersNickName +vVoO"
[16:23:14 CST(-0600)] * holdorph feels all registered and counted now. now to try to find something anonymous to do to balance things out.
[16:23:40 CST(-0600)] <EricDalquist> lol
[16:23:54 CST(-0600)] <holdorph> ok
[16:24:18 CST(-0600)] <holdorph> http://freenode.net/faq.shtml#registering
[16:24:54 CST(-0600)] <holdorph> in case you want to register lfuller, in particular the '/msg nickserv register <password> <email>' is all i did, then, the followup command it sends you in the email