Release Notes - Bigtop - Version 0.7.0

Sub-task

Bug

  • [BIGTOP-990] - /usr/lib/bigtop-utils/ should be owned by bigtop-utils package
  • [BIGTOP-998] - Consolidate Hue packages
  • [BIGTOP-1005] - Create versionless symlinks in hadoop client directory
  • [BIGTOP-1006] - Explicitly state the working directory to be used in init scripts instead of relying on user's home directories
  • [BIGTOP-1008] - /usr/lib/bigtop-utils directory is not owned by bigtop-utils package in RPMs
  • [BIGTOP-1010] - libsnappy not included in the final hadoop RPM
  • [BIGTOP-1011] - bigtop-detect-javahome has a quirky search order
  • [BIGTOP-1017] - Revert a commit that introduces some unneeded dependencies
  • [BIGTOP-1027] - 'bigtop-detect-classpath' helper script missing from bigtop-utils
  • [BIGTOP-1033] - itest-common: fix debian install to avoid conffile prompt
  • [BIGTOP-1034] - Custom user's home directory's creation fails in init-hdfs.sh
  • [BIGTOP-1036] - Hue's oozie app package should depend on hue jobsub package
  • [BIGTOP-1037] - Provide a mechanism to control the sourcing of defaults files
  • [BIGTOP-1044] - hive and hive-jdbc versions should be kept in sync
  • [BIGTOP-1045] - Be consistent with shells, bigtop-detect-javahome, and bigtop-utils versions
  • [BIGTOP-1046] - Pig's use of BIGTOP_DEFAULTS_DIR is evaluated at the wrong time
  • [BIGTOP-1051] - Spark archive should come in tar.gz format
  • [BIGTOP-1052] - Increase environment configurability/debugging of Mahout Tests
  • [BIGTOP-1055] - Spark package doesn't create PREFIX/bin directory and sets wrong permissions
  • [BIGTOP-1059] - Add path option for pig.jar to pig smoke pom.xml
  • [BIGTOP-1070] - Add BIGTOP_CLASSPATH functionality to tomcat deployment
  • [BIGTOP-1071] - bigtop-detect-javahome should select the first possible candidate in the list
  • [BIGTOP-1081] - deb spark packing needs to set SOURCE_DIR
  • [BIGTOP-1082] - spark package tests are missing
  • [BIGTOP-1084] - spark build on CentOS 5 and SLES11 is broken
  • [BIGTOP-1086] - fix miscellaneous failures in package tests
  • [BIGTOP-1087] - flume rpm package needs to own /usr/lib/flume/tools
  • [BIGTOP-1088] - oozie packages need to own /etc/oozie/conf.dist/tomcat-deployment.{default,secure}
  • [BIGTOP-1090] - hue packaging needs to handle mutable bits (app.reg and hue.pth)
  • [BIGTOP-1094] - spark SPEC mistakenly re-defines alternatives command on SLES/OpenSUSE
  • [BIGTOP-1095] - oozie packaging doesn't remove tomcat alternatives
  • [BIGTOP-1096] - alternatives within the alternatives-managed sub-directory could be harmful
  • [BIGTOP-1098] - now that Spark has had its first incubator release we have to adjust bigtop accordingly
  • [BIGTOP-1099] - sync up the versions used for smoke tests with the BOM versions
  • [BIGTOP-1100] - the new Tomcat deployment style would benefit from a few usability tweaks
  • [BIGTOP-1103] - Remaining fixes of BIGTOP_DEFAULTS_DIR evaluation
  • [BIGTOP-1107] - Update to the official Hue 2.5.1 release
  • [BIGTOP-1108] - Solr 4.5.0 has been formally released and we should start using the official release bits
  • [BIGTOP-1112] - sqoop-client should require bigtop-utils
  • [BIGTOP-1114] - Hue metastore app is missing from core registration and Debian packages
  • [BIGTOP-1115] - update package test manifest for Bigtop 0.7.0
  • [BIGTOP-1117] - hive-hcatalog integration has a few issues
  • [BIGTOP-1119] - phoenix DEB packages differ in a few ways from RPM counterparts

Improvement

  • [BIGTOP-927] - Update puppet recipes to support Ubuntu
  • [BIGTOP-938] - Upgrade to Junit 4.11
  • [BIGTOP-955] - HBase installation should advertise its location and configuration
  • [BIGTOP-1004] - Hadoop packages do not include version independent symlinks
  • [BIGTOP-1021] - Update HBase version to 0.94.9
  • [BIGTOP-1031] - README has outdated/ambiguous information
  • [BIGTOP-1041] - Add Puppet support for Spark cluster deployment
  • [BIGTOP-1056] - some changes are introduced into Spark build: needs to be updated here
  • [BIGTOP-1063] - Add Scala 2.9.3 to bigtop-toolchain
  • [BIGTOP-1069] - Remove descriptions about JDK5 dependency
  • [BIGTOP-1080] - Change /usr/bin scripts to be alternatives instead of flat files
  • [BIGTOP-1106] - update our puppet deployment topology

New Feature

  • [BIGTOP-811] - Add /var/lib/bigtop as a location to install SQL connectors and other plug-ins
  • [BIGTOP-1023] - Define BOM for 0.7.0 release of Bigtop

Task

Test

  • [BIGTOP-719] - Add TestTextSnappy to test hadoop fs -text with snappy compressed files
  • [BIGTOP-1030] - Develop integration tests for new Spark component
  • [BIGTOP-1057] - Add TeraGen / TeraSort Benchmakring

Release Notes - Bigtop - Version 0.6.0

Sub-task

  • [BIGTOP-239] - HBase init scripts are shut down in the wrong order
  • [BIGTOP-812] - HttpFS is using Tomcat 6.0.32 which has numerous vulnerabilities
  • [BIGTOP-813] - add integration test for HCatlog
  • [BIGTOP-814] - add packaging test for HCatlog
  • [BIGTOP-822] - Bump version of Hadoop to 2.0.5
  • [BIGTOP-824] - Bump version of Hive to 0.10
  • [BIGTOP-832] - Bump version of Hue to 2.3.0
  • [BIGTOP-833] - Update HCatalog packaging for Bigtop 0.6.0 integration
  • [BIGTOP-839] - add packaging test for Sqoop 2
  • [BIGTOP-852] - Improve HDFS helper script
  • [BIGTOP-854] - add package testing manifest for HCatalog
  • [BIGTOP-855] - add integration tests for HCatalog
  • [BIGTOP-861] - init-hdfs.sh does not work from init scripts
  • [BIGTOP-862] - All the Apache Hadoop services ought to start nicely and in order
  • [BIGTOP-863] - verify the existence of the initscripts with typo
  • [BIGTOP-864] - update the appliance for the location of the init-hdfs.sh script
  • [BIGTOP-880] - Bump version of Datafu to 0.0.6
  • [BIGTOP-891] - TestPigTest doesn't run against YARN cluster
  • [BIGTOP-915] - bump version of Oozie to 3.3.2
  • [BIGTOP-916] - Bump version of Pig to 0.11.1
  • [BIGTOP-917] - Bump version of Solr to 4.2.1
  • [BIGTOP-926] - Bump version of Giraph to 1.0
  • [BIGTOP-933] - Need to add ASF license text to bigtop-toolchain files
  • [BIGTOP-962] - Bump Sqoop version to 1.99.2
  • [BIGTOP-963] - bump version of Whirr to 0.8.2
  • [BIGTOP-980] - bigtop-toolchaing user module can't create 'jenkins' user

Bug

  • [BIGTOP-275] - scoping of variable in puppet code is incorrect
  • [BIGTOP-379] - Package testing for Bigtop 0.5.0 release
  • [BIGTOP-445] - yarn default file has the wrong name
  • [BIGTOP-463] - should we reconsider /usr/lib vs. /usr/libexec decision?
  • [BIGTOP-483] - Smoke test of Hadoop fails with clitest missing testConf.xml file
  • [BIGTOP-489] - TestCLI with two test cases fails ID - 146 and 147 related to refreshServiceAcl
  • [BIGTOP-504] - provide a functionality for detecting libjvm.so in bigtop-utils
  • [BIGTOP-545] - package testing manifest in trunk needs to be updated
  • [BIGTOP-553] - Metadata for packages needs to be harmonized between RPM and Debain
  • [BIGTOP-584] - Fix hive smoke test failures in trunk
  • [BIGTOP-624] - running start on a service already running should return success
  • [BIGTOP-637] - Update boxgrinder appliance for the coming release
  • [BIGTOP-786] - Not able to find wiki link on Bigtop homepage
  • [BIGTOP-804] - org.apache.bigtop.itest.pmanager.PackageManagerTest.testLookupGcc fails on some systems
  • [BIGTOP-808] - hadoop-conf-pseudo package breaks hadoop-conf alternatives during upgrades
  • [BIGTOP-809] - Remove harmless scary message from solr build
  • [BIGTOP-815] - Move the trunk forward now that 0.5.0 is released
  • [BIGTOP-818] - Updated list of supported platforms on Bigtop's homepage and readme file
  • [BIGTOP-823] - Remove obsolete slf4j removal in hive packaging
  • [BIGTOP-826] - Wrong return code returned when stopping Hue on RHEL/SUSE
  • [BIGTOP-827] - Ubuntu Install Instructutions Fail with a 404
  • [BIGTOP-829] - Hue status returned wrong information on SLES
  • [BIGTOP-830] - Add $(APACHE_MIRROR) for bigtop-tomcat
  • [BIGTOP-834] - Remove workaround for MAPREDUCE-4814 in historyserver
  • [BIGTOP-835] - The shell exec method must have variants which have timeout and can run in background
  • [BIGTOP-836] - Hue status on SuSE doesn't work when run from outside of /usr/bin
  • [BIGTOP-837] - Oozie 3.3.0 can't be build against Hadoop 2.0.3 or later
  • [BIGTOP-841] - hadoop-conf-pseudo missed configuration for capacity scheduler and historyserver
  • [BIGTOP-842] - hadoop-mapreduce-historyserver service dumps some output to the console instead of its .out file
  • [BIGTOP-844] - hadoop rpm upgrade sequence is broken
  • [BIGTOP-846] - The helper script introduced in BIGTOP-547 doesn't create /user/$USER directories on HDFS
  • [BIGTOP-850] - Remove non-ASF repositories and infrastructure references from the project files
  • [BIGTOP-851] - Typo breaks detection of multiple libraries in bigtop-utils in some JVMs
  • [BIGTOP-853] - HBase test, TestLoadAndVerify does not work on secure clusters
  • [BIGTOP-857] - Inconsistent group ownership of /var/run/hadoop-hdfs between .deb and .rpm installs
  • [BIGTOP-858] - HBase tests are no longer compile with HBase 0.94
  • [BIGTOP-859] - fix the Bigtop trunk package test
  • [BIGTOP-869] - hadoop preun should not stop services itself on behalf of its subpackages
  • [BIGTOP-870] - Bump version of Pig to 0.11
  • [BIGTOP-873] - Hive needs Ivy cache cleared
  • [BIGTOP-874] - make test contracts show up in javadoc
  • [BIGTOP-875] - hadoop-smoke is broken because of missed dependency
  • [BIGTOP-876] - TestDFSAdmin in hadoop-smoke is outdated and still relies on presence of -upgradeProgress
  • [BIGTOP-877] - TestDFSAdmin in hadoop-smoke has to ensure its own environment, e.g. create HDFS paths, etc.
  • [BIGTOP-878] - Bump version of Oozie to 3.3.1
  • [BIGTOP-879] - Bump version of Solr to 4.2.0
  • [BIGTOP-883] - Add support for Oracle RPM Java 1.7
  • [BIGTOP-884] - hive smoke tests can't be executed as a jar file isn't present anymore
  • [BIGTOP-885] - TestHiveSmokeBulk fails on Hive 0.9
  • [BIGTOP-886] - Non-fatal errors when starting Hive daemons in EL5
  • [BIGTOP-887] - Sqoop 2 should be consistent with Oozie / Bigtop for server plugins directory
  • [BIGTOP-888] - Oozie losing value of JAVA_HOME when spawning sub-shell
  • [BIGTOP-889] - pig smoke tests are no longer running
  • [BIGTOP-890] - Remove workaround for distributed cache problem from Oozie package
  • [BIGTOP-892] - hadoop packages misses capacity-scheduler configuration
  • [BIGTOP-894] - Pig compilation fails on RPM systems on Bigtop trunk
  • [BIGTOP-895] - A number of testcases in TestCLI are failing with (at least) Hadoop 2.0.3 and later
  • [BIGTOP-896] - Solr rpm requires redhat-lsb to run
  • [BIGTOP-899] - enabled deployment topology with 3 nodes
  • [BIGTOP-900] - update deployment configs for Kerberized installations
  • [BIGTOP-901] - Oozie smoke tests mightn't be able to obtain the jobtracker or namenode hostnames, using old config. names
  • [BIGTOP-902] - Hive uses same file for log4j and standard output
  • [BIGTOP-903] - crunch integration tests need to clean up output directory before each individual subtask is run
  • [BIGTOP-904] - mahout tests should default to /tmp as a temp location
  • [BIGTOP-905] - bump versions of test execution poms to match the BOM
  • [BIGTOP-906] - a few tests need artifacts compiled against Hadoop 2.X
  • [BIGTOP-907] - hadoop-mapreduce-historyserver should require hadoop-hdfs
  • [BIGTOP-910] - datanode init script cannot stop the process bcause it cannot find the pid file
  • [BIGTOP-912] - update HBase tests for 0.6.0
  • [BIGTOP-914] - puppet needs to provision test users
  • [BIGTOP-918] - Move BOM version info to the root pom file
  • [BIGTOP-920] - Add HCatalog tests
  • [BIGTOP-923] - Packages that depend on bigtop-utils needs to specify >= 0.6
  • [BIGTOP-928] - Hive can't be build against non-published Hadoop artifacts
  • [BIGTOP-929] - Pig component build needs to use internal resolvers
  • [BIGTOP-930] - init-hdfs.sh fails on execution
  • [BIGTOP-934] - Hidden cookie is being set by wiki.cloudera.org via http://blogs.apache.org/bigtop top page
  • [BIGTOP-935] - Use init script templates for hive
  • [BIGTOP-936] - Use init script templates for hbase
  • [BIGTOP-937] - Hive does not restart on Ubuntu
  • [BIGTOP-940] - fix issues with Hive package
  • [BIGTOP-945] - Service lock files need to match init script name
  • [BIGTOP-946] - solr-server runlevels are wrong in debian
  • [BIGTOP-947] - oozie init script does not use JAVA_HOME in /etc/default/bigtop-utils
  • [BIGTOP-948] - TestFileAppend and TestHDFSQuota tests need to be updated
  • [BIGTOP-949] - Add Sqoop tests
  • [BIGTOP-950] - race condition for output consumption in Shell code
  • [BIGTOP-951] - Hive Smoke's TestJdbcDriver fails
  • [BIGTOP-953] - Revert BIGTOP-835 and BIGTOP-950
  • [BIGTOP-954] - metadada and user data for sqoop and hcatalog packages are wrong
  • [BIGTOP-956] - fix small issues with mahout, hue and oozie tests
  • [BIGTOP-957] - update deployment code for Sqoop2
  • [BIGTOP-958] - update deployment code for HCatalog
  • [BIGTOP-959] - get rid of hadoop specific init.d.tmpl
  • [BIGTOP-960] - Remove workaround for HBASE-6263 in hbase-thrift start
  • [BIGTOP-968] - oozie is now part of the core hue
  • [BIGTOP-969] - a few files belong to more than one package
  • [BIGTOP-970] - update package test manifest and fix small issues in package metadata for Bigtop 0.6.0
  • [BIGTOP-971] - take care of the rat licensing complaints
  • [BIGTOP-974] - Crunch download location is no longer /incubator
  • [BIGTOP-975] - HBase regionserver init script does not implement "condrestart" command
  • [BIGTOP-981] - Provide minimal documentation for puppet deploy config file
  • [BIGTOP-982] - hcatalog needs to be given HCAT_HOME and HIVE_HOME
  • [BIGTOP-983] - Oozie-client package ships its docs twice
  • [BIGTOP-984] - bigtop-utils doesn't detect JAVA_HOME for Oracle JDK 7 when package by Debian's tool
  • [BIGTOP-985] - Create Hive JDBC package
  • [BIGTOP-988] - HCatalog does not launch daemon process in a known directory
  • [BIGTOP-991] - crunch archive is incorrect
  • [BIGTOP-994] - hdfs-init missed a few permissions on the sub-directories
  • [BIGTOP-997] - TestCLI fails test id 5: comparator fails
  • [BIGTOP-1001] - need package tests for hive-jdbc package
  • [BIGTOP-1002] - Complete porting Sqoop import tests to Sqoop 2

Improvement

  • [BIGTOP-12] - Add HCatalog to Bigtop
  • [BIGTOP-547] - create an HDFS setup helper script
  • [BIGTOP-712] - Bigtop 0.5.0 release
  • [BIGTOP-713] - use newer debhelper and source format 3.0 (quilt) for Debian and Ubuntu packaging
  • [BIGTOP-756] - Provide the thrift API interface file in the HBase packages
  • [BIGTOP-763] - Add hdfs-journalnode to hadoop.spec
  • [BIGTOP-817] - Wiki breadcrumb still shows Bigtop(incubating)
  • [BIGTOP-821] - Puppet README's Usage section cuts off because of angle bracket delimiter
  • [BIGTOP-838] - Both conf.empty and conf.pseudo packages don't seem to include a sample hadoop-env.sh file
  • [BIGTOP-843] - It would be really cool if bigtop-detect-javahome were smarter and didn't pick a 32-bit JVM, or a 1.7 JVM set with default
  • [BIGTOP-882] - Upload content of Oozie sharelib to HDFS
  • [BIGTOP-893] - migrate puppet code to use init-hdfs.sh
  • [BIGTOP-931] - a few improvements to secure puppet deployment code
  • [BIGTOP-961] - hcatalog should be re-using hive user when running
  • [BIGTOP-967] - package removal is currently missing from our package tests
  • [BIGTOP-973] - Improve error diagnostics when sed'ing Hadoop version in Oozie package
  • [BIGTOP-995] - JDK bug 6979329 requires a redundant kinit -R
  • [BIGTOP-1000] - Remove remaining references to the Incubator

New Feature

  • [BIGTOP-732] - Support running multiple HBase region servers
  • [BIGTOP-881] - Add support for Oozie to be configured with SSL (HTTPS)

Task

  • [BIGTOP-118] - Check to ensure our build system requirements are actually declared properly in the packaging.
  • [BIGTOP-681] - Package tests needed for new DataFu package
  • [BIGTOP-720] - Build environment automation
  • [BIGTOP-760] - Push artifacts from non-Maven projects into local Maven cache
  • [BIGTOP-805] - Add Sqoop 2 to bigtop

Test

Release Notes - Bigtop - Version 0.5.0

Sub-task

  • [BIGTOP-39] - Include Apache Avro in Bigtop
  • [BIGTOP-41] - Add Apache Thrift to Bigtop
  • [BIGTOP-65] - Package Apache Forrest
  • [BIGTOP-225] - Exit consistently in init scripts for stop on stopped service
  • [BIGTOP-232] - create an example .ks file for Bigtop
  • [BIGTOP-237] - Oozie init scripts are way too verbose
  • [BIGTOP-239] - HBase init scripts are shut down in the wrong order
  • [BIGTOP-240] - hbase initscripts should be more explicit
  • [BIGTOP-302] - 1. Start a separate Wiki page on Installing Bigtop in VMs (VirtualBox) (make sure to record the fact that one needs >> 2GB of RAM) 2. Record the experience with 32bit OSes
  • [BIGTOP-380] - Initscripts have misleading messages
  • [BIGTOP-533] - BigTop Hama Integration Tests
  • [BIGTOP-551] - Updates for packaging manifest
  • [BIGTOP-612] - Add Crunch to Bigtop
  • [BIGTOP-692] - Bump Whirr to 0.8.1
  • [BIGTOP-695] - Make variables used in flume build/install scripts consistent
  • [BIGTOP-696] - Make variables used in oozie build/install scripts consistent
  • [BIGTOP-697] - Make variables used in zookeeper build/install scripts consistent
  • [BIGTOP-700] - Make variables used in whirr build/install scripts consistent
  • [BIGTOP-702] - Make variables used in datafu build/install scripts consistent
  • [BIGTOP-703] - Make variables used in jsvc build/install scripts consistent
  • [BIGTOP-704] - Make variables used in pig build/install scripts consistent
  • [BIGTOP-705] - Make variables used in giraph build/install scripts consistent
  • [BIGTOP-706] - Make variables used in hive build/install scripts consistent
  • [BIGTOP-707] - Make variables used in tomcat build/install scripts consistent
  • [BIGTOP-711] - update hadoop version to 2.0.2-alpha
  • [BIGTOP-714] - HDFS Fuse build broken for 0.23.3
  • [BIGTOP-733] - Bump Hue to 2.1.0
  • [BIGTOP-735] - Upgrade to pull Solr 4.0 when it's officially released
  • [BIGTOP-736] - When Solr is integrated, we'll need some tests.
  • [BIGTOP-744] - add package test for Solr
  • [BIGTOP-750] - Bump Zookeeper to 3.4.5
  • [BIGTOP-751] - Bump HBase to 0.94.2
  • [BIGTOP-752] - Bump Sqoop to 1.4.2
  • [BIGTOP-753] - Bump Flume to 1.3.0
  • [BIGTOP-765] - Create deployment code for Solr
  • [BIGTOP-791] - Bump Oozie to 3.3.0

Bug

  • [BIGTOP-51] - Layout of Bigtop repository can be optimized
  • [BIGTOP-79] - Audit RPM package dependencies for programs invoked from install/uninstall scripts
  • [BIGTOP-98] - Ability to force ivy/maven version inter-dependency needs to be implemented
  • [BIGTOP-105] - hadoop services can not be started with the default configs in place
  • [BIGTOP-106] - review and cleanup install/unistall/upgrade time dependencies
  • [BIGTOP-107] - make Bigtop packaging work in kick-start environment
  • [BIGTOP-135] - make jsvc use detection more robust on SLES
  • [BIGTOP-168] - all of the packages that link to hadoop-core need to also link commons-configuration-1.6.jar
  • [BIGTOP-172] - Hadoop init.d scripts are flaky on stop
  • [BIGTOP-177] - Remove or fix versioned dependencies between packages of Bigtop
  • [BIGTOP-187] - some packages ship log files
  • [BIGTOP-200] - /var/lib/hadoop is set to 777 on installation it should just be group writable
  • [BIGTOP-213] - We should use another group than hadoop for system groups
  • [BIGTOP-218] - we need to figure out which compressors/codecs need to be pulled into Bigtop
  • [BIGTOP-219] - Pig/HBase integration is not working
  • [BIGTOP-222] - some man pages in /usr/share/doc/flume
  • [BIGTOP-223] - install/ initscripts fixes
  • [BIGTOP-224] - Some configuration files have their permissions set to 755
  • [BIGTOP-226] - everything in /usr/lib/pig is set to 755. Non executable files should just be 644
  • [BIGTOP-227] - whirr has all its files (except its man page) set as 755
  • [BIGTOP-228] - Whirr should have a symlink to the hadoop's jars it needs
  • [BIGTOP-229] - pig package install an alternative with an inconsitent name
  • [BIGTOP-230] - removing packages displays the following extraneous messages
  • [BIGTOP-256] - Lintian Errors/Warnings - Hadoop Package
  • [BIGTOP-275] - scoping of variable in puppet code is incorrect
  • [BIGTOP-279] - we shouldn't be using properties that have been deprecated in config/puppet
  • [BIGTOP-282] - the licensing status of the MovieLens data files needs to be cleared up
  • [BIGTOP-292] - BIgtop SRPMs can not be rebuilt
  • [BIGTOP-307] - README documentation is inaccurate
  • [BIGTOP-311] - hadoop does not build in off-line mode
  • [BIGTOP-326] - puppet modules need to be renamed to align with renaming that happens on the package front
  • [BIGTOP-330] - hadoop 0.23 pseudo conf needs to set more properties to avoid using /tmp as its datadir
  • [BIGTOP-335] - sqoop-metastore service fails to start because it ends up using /usr/lib/hadoop/bin/hadoop
  • [BIGTOP-340] - wrapper scripts would benefit from being explicit
  • [BIGTOP-347] - we need to get rid of workaround for HDFS-1943 in hadoop init.d scripts
  • [BIGTOP-358] - now that hadoop packages have been split we have to update the dependencies on the downstream packages
  • [BIGTOP-368] - /usr/bin/hadoop sources /etc/default/yarn unconditionally
  • [BIGTOP-377] - Clean up packages
  • [BIGTOP-379] - Package testing for Bigtop 0.5.0 release
  • [BIGTOP-385] - new artifacts in Hadoop 0.23 need to be properly packaged
  • [BIGTOP-395] - flume-ng init.d scripts seem to be unhappy on Debian
  • [BIGTOP-425] - HMaster dies from RemoteException: Server IPC version 5 cannot communicate with client version 4
  • [BIGTOP-436] - flume-node stop seems to mistarget some other java process on lucid
  • [BIGTOP-445] - yarn default file has the wrong name
  • [BIGTOP-463] - should we reconsider /usr/lib vs. /usr/libexec decision?
  • [BIGTOP-471] - hive server setup after installation fails on creating /var/lock/subsys directory
  • [BIGTOP-483] - Smoke test of Hadoop fails with clitest missing testConf.xml file
  • [BIGTOP-486] - separate principals from keytabs in our kerberos module
  • [BIGTOP-489] - TestCLI with two test cases fails ID - 146 and 147 related to refreshServiceAcl
  • [BIGTOP-490] - SmokeTest-Mahout test failures
  • [BIGTOP-492] - make our launcher scripts recognize cascading defaults
  • [BIGTOP-504] - provide a functionality for detecting libjvm.so in bigtop-utils
  • [BIGTOP-511] - Add debian repo to bigtop puppet
  • [BIGTOP-520] - HBase Debian builds run do-component-build twice
  • [BIGTOP-525] - make sure yarn.dispatcher.exit-on-error is set to true
  • [BIGTOP-531] - package metadata check error: when description is empty, the summary will append text "description" at the end of summary
  • [BIGTOP-542] - Issue when installing the new zkfc
  • [BIGTOP-545] - package testing manifest in trunk needs to be updated
  • [BIGTOP-549] - update puppet code for the latest state of HA in Hadoop
  • [BIGTOP-553] - Metadata for packages needs to be harmonized between RPM and Debain
  • [BIGTOP-566] - Flume NG pkg init script should allow user to customize the location of the conf dir
  • [BIGTOP-581] - Need to move iTest website to a correct location
  • [BIGTOP-584] - Fix hive smoke test failures in trunk
  • [BIGTOP-591] - libhdfs0-dev is missing files
  • [BIGTOP-615] - Add support for Ambrose Pig job visualization
  • [BIGTOP-624] - running start on a service already running should return success
  • [BIGTOP-634] - Weird output when doing an init action on oozie when it is already running
  • [BIGTOP-637] - Update boxgrinder appliance for the coming release
  • [BIGTOP-644] - fix the runlevels for a few packages in yum/apt/zypper
  • [BIGTOP-648] - hbase-thrift cannot be started properly
  • [BIGTOP-654] - Set nproc for HDFS user
  • [BIGTOP-658] - Move hbase dependency in hive out of the main package
  • [BIGTOP-662] - cannot start hue-server
  • [BIGTOP-663] - cannot start sqoop-metastore in Ubuntu precise
  • [BIGTOP-664] - hive-metastore script has error when execute 'service hive-metastore status' in Ubuntu precise
  • [BIGTOP-667] - start hadoop-httpfs in SLES return 3 instead of 0
  • [BIGTOP-669] - Add DataFu to Bigtop distribution
  • [BIGTOP-676] - Update HBase version to 0.94.1-rc0
  • [BIGTOP-679] - fuse_dfs binary has moved due to HDFS-3537
  • [BIGTOP-684] - copyright header in bigtop-packages/src/common/hadoop/conf.secure/core-site.xml is in wrong place
  • [BIGTOP-686] - Apache Flume 1.2.0 went out so we should upgrade to it
  • [BIGTOP-689] - Oozie use of /var/tmp in some CentOS versions create issues
  • [BIGTOP-691] - flume gets killed too fast when the service is asked to stop
  • [BIGTOP-694] - Some HBase Scripts Need To Be Excluded From Packaging
  • [BIGTOP-699] - hadoop-fuse-dfs fails to find libjvm.so when JAVA_HOME happens to be a symlink
  • [BIGTOP-701] - Bigtop Hadoop package does not include bash tab completion
  • [BIGTOP-708] - modify Hive integration test to depends on the correct jar file
  • [BIGTOP-721] - improve the package daemon status check, check twice by some delay if status doesn't match expected value
  • [BIGTOP-725] - package service test TestServices should only do config once for a group of services
  • [BIGTOP-729] - zookeeper initd script status doesn't work
  • [BIGTOP-734] - Proposed fixes to Hive logging configuration
  • [BIGTOP-740] - improve the package file content tests to ignore platform specific file names
  • [BIGTOP-745] - fix incorrect javadocs in test framework
  • [BIGTOP-754] - libhdfs0-dev package is empty
  • [BIGTOP-755] - YARN_HOME is now HADOOP_YARN_HOME
  • [BIGTOP-758] - [Pig] Override hadoop-core and hadoop-test artifact versions only if Hadoop < 2.x
  • [BIGTOP-759] - Flume do-component-build should set hbase.version according to BOM
  • [BIGTOP-762] - Updates for building Flume versions >= 1.2
  • [BIGTOP-766] - Make it possible to override Apache URLs for the builds that require them
  • [BIGTOP-768] - Bump the version of Groovy to 1.8
  • [BIGTOP-774] - solr build fails on Debian
  • [BIGTOP-775] - In Groovy 1.8 the behavior of a getters of static fields changed slightly
  • [BIGTOP-776] - A few updates to Oozie related packaging code to accomodate Oozie's TLP status
  • [BIGTOP-778] - move Solr app to under /solr in Tomcat deployment and provide more knobs
  • [BIGTOP-780] - cannot install hive-server in SLES
  • [BIGTOP-781] - Bigtop packages seem to have a bunch of parasitic provides
  • [BIGTOP-782] - 'service hue status' still show 'failed' after hue is started in SLES
  • [BIGTOP-783] - 'service hue stop' cannot stop the hue service in SLES
  • [BIGTOP-785] - clean up build-time dependencies in our packages
  • [BIGTOP-788] - Oozie RPM build error due to lack of usr/lib/oozie/{libserver,oozie-server,libext}
  • [BIGTOP-789] - remove shadow-utils dependency from zookeeper
  • [BIGTOP-793] - add crunch integration tests
  • [BIGTOP-794] - puppet code around hadoop_head_node breaks on newer puppets
  • [BIGTOP-795] - need to abstract away the notion of Bigtop puppet repo
  • [BIGTOP-796] - hive install tries to tickle HDFS and it shouldn't
  • [BIGTOP-797] - provide a way to 'rsync' files into HDFS during puppet deployment
  • [BIGTOP-798] - introduce a fatjar collection of all the Bigtop integration tests and all their dependencies
  • [BIGTOP-800] - BIGTOP_JSVC is lacking ARCHIVE designation
  • [BIGTOP-801] - prioritize jdk over jre in bigtop-utils javadetection
  • [BIGTOP-803] - somehow puppet gets confused by the hadoop fs output

Improvement

  • [BIGTOP-12] - Add HCatalog to Bigtop
  • [BIGTOP-22] - Unify init scripts
  • [BIGTOP-58] - Add man page for Mahout
  • [BIGTOP-60] - Add normal rpmbuild options for make rpm
  • [BIGTOP-78] - Misc improvments
  • [BIGTOP-117] - Document incompatible API changes between releases
  • [BIGTOP-153] - Build should detect invalid archives in the dl directory
  • [BIGTOP-204] - Add information about supported projects and criteria for support to Apache Bigtop website
  • [BIGTOP-233] - Ubuntu integration: ufw application profiles
  • [BIGTOP-245] - Improve Bigtop layout
  • [BIGTOP-263] - we need a way to provide supervision for long running daemon applications
  • [BIGTOP-284] - It would be nice to integrate Nutch crawling as one of the MR (and perhaps HBase) tests for Bigtop
  • [BIGTOP-304] - document test creation and test execution steps in Bigtop
  • [BIGTOP-351] - Make HBASE_HEAPSIZE manageable via puppet
  • [BIGTOP-375] - Publish a table of component dependency versions
  • [BIGTOP-410] - Build secure version of HBase
  • [BIGTOP-433] - Make variables used in build/install scripts consistent
  • [BIGTOP-451] - ooziedb.sh script should run as the unix user that runs oozie server
  • [BIGTOP-452] - oozie installation should offer to run ooziedb.sh
  • [BIGTOP-456] - Consider splitting homedir between mapred and hdfs users?
  • [BIGTOP-476] - Improvement of BigTop iTest framework
  • [BIGTOP-497] - add bash tab completion to our hadoop packages
  • [BIGTOP-517] - Flume packaging should use the term "agent" instead of "node".
  • [BIGTOP-536] - provide suitable title in the <title> tag of the main home page for bigtop-incubating
  • [BIGTOP-547] - create an HDFS setup helper script
  • [BIGTOP-550] - Add ability to have multiple zookeeper servers to puppet
  • [BIGTOP-571] - we might need to guard against direct use of the upstream launcher scripts
  • [BIGTOP-575] - [Pseudo-dist config] Should yarn.nodemanager.remote-app-log-dir really be set?
  • [BIGTOP-576] - Minor discrepancy in naming of logs/users vs. base log directories
  • [BIGTOP-640] - Sync up dependency library versions automatically
  • [BIGTOP-670] - provide a link to our Jenkins server from the Bigtop's project page
  • [BIGTOP-680] - Write util function to capture test input parameters
  • [BIGTOP-688] - improve hue packaging via making virtual env relocatable and moving DB files into /var/lib/hue
  • [BIGTOP-710] - Create a higher level orchestration deployment framework
  • [BIGTOP-712] - Bigtop 0.5.0 release
  • [BIGTOP-713] - use newer debhelper and source format 3.0 (quilt) for Debian and Ubuntu packaging
  • [BIGTOP-716] - standardize on bash as a shell of choice
  • [BIGTOP-718] - Update wiki for component requirements
  • [BIGTOP-726] - make it possible to run HiveBulkScriptExecutor scripts selectively
  • [BIGTOP-731] - Reorganize Hadoop tests
  • [BIGTOP-742] - tighten permissions on hadoop services home directories
  • [BIGTOP-756] - Provide the thrift API interface file in the HBase packages
  • [BIGTOP-757] - Install ZooKeeper jars into the local Maven cache
  • [BIGTOP-769] - Create a generic shell executor iTest driver
  • [BIGTOP-772] - Remove all references to the incubation from the trunk of Bigtop
  • [BIGTOP-777] - separate the Package Tests into service tests, file content tests, and dependency tests
  • [BIGTOP-779] - create a config management utility for SolrCloud
  • [BIGTOP-784] - test deployment code on all of the supported platforms (using OpenJDK7 where available)
  • [BIGTOP-787] - Surface all possible options in /etc/default/flume-node
  • [BIGTOP-792] - org.apache.bigtop.itest.posix.Service needs to be updated to accomodate upstart/systemd
  • [BIGTOP-799] - create an itest method for unpacking all the resources into the HDFS or local filesystem
  • [BIGTOP-802] - Add rollback option to DataNode service script
  • [BIGTOP-806] - now that MAPREDUCE-3916 is fixed we should enable webproxy

New Feature

  • [BIGTOP-42] - Start doing testpatch-like patch validation
  • [BIGTOP-64] - Package Apache tools used in Bigtop builds
  • [BIGTOP-101] - Need to come up with a strategy for dealing with soft dependencies between packages
  • [BIGTOP-103] - add support for upstart service management facility
  • [BIGTOP-116] - Add support for systemd
  • [BIGTOP-220] - Add mrunit to Bigtop
  • [BIGTOP-246] - Provide more verbose output at artifact build time
  • [BIGTOP-276] - Standardize how clients create classpaths and java.library.paths
  • [BIGTOP-350] - Some sort of system expert would be useful for manifests reviews
  • [BIGTOP-399] - Provide a one-click install package for seamless deployment of Bigtop from the release/nightly repos
  • [BIGTOP-458] - Create some builds/tests jobs for openJDK 6 and 7 on bigtop01 jenkins
  • [BIGTOP-460] - Add support for daemontools-encore
  • [BIGTOP-524] - Add tests to verify that our init scripts conform to the LSB specification
  • [BIGTOP-594] - Introduce git-style bigtop command line and BIGTOP_HOME environment variable
  • [BIGTOP-601] - introduce bigtop-build packages for packaging build-time dependencies
  • [BIGTOP-635] - Implement a cluster-abstraction, discovery and manipulation framework for iTest
  • [BIGTOP-685] - provide a way to specify the parameters expected by a test
  • [BIGTOP-687] - Support YARN as a subproject of Hadoop
  • [BIGTOP-730] - add tarball driver into Bigtop

Task

  • [BIGTOP-236] - The user installed by hbase gets an incorrect home directory
  • [BIGTOP-314] - add ZooKeeper systest to Bigtop testing regime
  • [BIGTOP-406] - Enable openSuse VM on bigtop01
  • [BIGTOP-439] - we need to review package test manifest for the Bigtop 0.4.0 release
  • [BIGTOP-681] - Package tests needed for new DataFu package
  • [BIGTOP-715] - Add Spark packaging
  • [BIGTOP-720] - Build environment automation
  • [BIGTOP-760] - Push artifacts from non-Maven projects into local Maven cache

Test

Wish

  • [BIGTOP-67] - Resize bigtop logo in header of website

Release Notes - Bigtop - Version 0.4.0

Sub-task

  • [BIGTOP-249] - Fix namenode webapp
  • [BIGTOP-250] - Build contrib modules
  • [BIGTOP-251] - Making sure security works with the new packages for hadoop 0.23
  • [BIGTOP-254] - Write init scripts for the new yarn services
  • [BIGTOP-268] - Examples do not run
  • [BIGTOP-270] - Need to create versionless jars for integration point with other projects
  • [BIGTOP-271] - Kill the sbin package for hadoop 0.23
  • [BIGTOP-299] - Recording the fact that Java needs to be installed *before* installing Bigtop (at least on Ubuntu)
  • [BIGTOP-300] - Record the information on slow mirrors
  • [BIGTOP-387] - Zookeeper version needs to be updated to 3.4.3
  • [BIGTOP-424] - Bump version of Oozie to 3.1.3
  • [BIGTOP-527] - Integrate Hue 2.0.0 into Bigtop
  • [BIGTOP-538] - update puppet code for BIGTOP-535
  • [BIGTOP-539] - update puppet code for BIGTOP-534
  • [BIGTOP-603] - Bump version of Pig to 0.10.0
  • [BIGTOP-604] - Bump version of Hive to 0.9.0
  • [BIGTOP-605] - Create How To Release document
  • [BIGTOP-606] - Bump version of HBase to 0.94.0
  • [BIGTOP-617] - Bump version of Oozie to 3.2.0
  • [BIGTOP-628] - bump version of Mahout to 0.7.0

Bug

  • [BIGTOP-96] - Mahout examples try to write to /usr/lib/mahout/examples/bin/work
  • [BIGTOP-162] - downstream components need to be patched to be compatible with MR2
  • [BIGTOP-186] - whirr package ships source code and needs some permissions fixes
  • [BIGTOP-264] - Hadoop 23 upstream tarball put some of its hadoop jars in the same directory as some of the dependencies
  • [BIGTOP-269] - in .23 HBase need to link to a much wider set of hadoop jars
  • [BIGTOP-272] - small cleanup for hadoop 0.23
  • [BIGTOP-274] - yarn-site.xml needs to be addedd to the Hadoop client classpath
  • [BIGTOP-277] - hive package in hadoop-0.23 branch is hopelessly broken
  • [BIGTOP-283] - in hadoop-0.23 the sha for Hadoop needs to be bumped to the 23.1-SNAPSHOT cut off point
  • [BIGTOP-285] - Remove source jars from hadoop-0.23
  • [BIGTOP-286] - hadoop-0.23: hadoop-mapreduce-examples.jar is empty
  • [BIGTOP-288] - hadoop-0.23: pig contrib should have jars
  • [BIGTOP-289] - hadoop-0.23: need to package native libraries for compression
  • [BIGTOP-291] - __id_u RPM macro seems to be disabled in newer versions of RPMs
  • [BIGTOP-294] - brp-repack-jars chokes on guice-2.0.jar because of the directories without any read permission
  • [BIGTOP-308] - hadoop-0.23 branch doesn't provide .so links to hadoop native libraries
  • [BIGTOP-309] - hadoop-0.23 puppet code needs to deploy history server
  • [BIGTOP-312] - Puppet resource dependency syntax used in hadoop/manifests/init.pp doesn't always work
  • [BIGTOP-316] - split up hadoop packages into common, hdfs, mapreduce (and yarn)
  • [BIGTOP-325] - bump version of Zookeeper to 3.4.2
  • [BIGTOP-331] - our init scripts fail to detect a failed history server in hadoop 0.23
  • [BIGTOP-333] - heed to start properly packaging httpfs
  • [BIGTOP-338] - hadoop 0.23 doc package does not contain generated documentation
  • [BIGTOP-339] - zookeeper init.d script has a typo in PID file location
  • [BIGTOP-341] - hadoop streaming is not available in hadoop-0.23 branch
  • [BIGTOP-344] - puppet code needs to be updated to be able to deploy Hadoop 0.23 YARN in a secure fashion
  • [BIGTOP-345] - container-executor.conf.dir needs to be set to /etc/hadoop/conf
  • [BIGTOP-346] - container-executor needs to be owned by root:yarn and have permissions of 6050
  • [BIGTOP-348] - jsvc location needs to be updated in init.d datanode script
  • [BIGTOP-349] - jsvc from hadoop build fails on some of the Linux systems that Bigtop has to support
  • [BIGTOP-355] - missed yarn.nodemanager.local-dirs, yarn.nodemanager.log-dirs and yarn.nodemanager.remote-app-log-dir in our puppet code
  • [BIGTOP-357] - libjvm.so dependency on CentOS can only be satisfied by libgcj in some cases
  • [BIGTOP-359] - documentation build is picked from the wrong place
  • [BIGTOP-367] - Upgrade sequencing issue
  • [BIGTOP-369] - pig package has license files in the wrong location
  • [BIGTOP-372] - hive can't be started without setting HADOOP_HOME
  • [BIGTOP-373] - bigtop-detect-javahome not customizable
  • [BIGTOP-376] - Remove unused obsolete statement in hadoop packaging
  • [BIGTOP-378] - history server security configs are missing
  • [BIGTOP-381] - need to catch up with the HEAD of hadoop-common in hadoop-0.23
  • [BIGTOP-383] - bigtop-detect-javahome installed at location unexpected by scripts on SLES
  • [BIGTOP-389] - need to start packaging apache commons daemon jsvc
  • [BIGTOP-392] - zookeeper package needs to be updated for 3.4 series
  • [BIGTOP-393] - hadoop packages don't set nproc
  • [BIGTOP-396] - Missing resource dependencies in puppet for secure clusters
  • [BIGTOP-397] - hbase, sqoop and oozie don't use alternatives for managing configuration
  • [BIGTOP-402] - manifests for package testing contain way too many duplicate data
  • [BIGTOP-407] - hadoop package can't be installed on debian
  • [BIGTOP-408] - HBase build needs to be worked around in hadoop-0.23 branch
  • [BIGTOP-409] - sqoop build fails on SLES
  • [BIGTOP-415] - need to workaround MAPREDUCE-3890
  • [BIGTOP-416] - run levels for init.d scripts need to be harmonized
  • [BIGTOP-417] - webapps shipped by bigtop components should be using a system-provided Tomcat
  • [BIGTOP-418] - Package MAPREDUCE-2858 (MRv2 WebApp Security)
  • [BIGTOP-420] - bigtop-detect-javahome installed at location unexpected by scripts on SLES
  • [BIGTOP-421] - Sign every single file from the convenience artefact in the releases
  • [BIGTOP-422] - need to start packaging fuse for Hadoop 0.23
  • [BIGTOP-423] - hadoop package needs to be split into hadoop-client and hadoop-server packages
  • [BIGTOP-426] - build failures in hadoop-0.23 branch
  • [BIGTOP-428] - sqoop build needs to be forced to depend on HBase 0.92.0 artifacts
  • [BIGTOP-429] - CATALINA_PID needs to be updated in hadoop-httpfs.default to correspond to the calculated value
  • [BIGTOP-430] - oozie examples are missing
  • [BIGTOP-434] - Potential issue in the way Bigtop test infrastructure deal with arch in packages
  • [BIGTOP-435] - need to split hadoop subpackages even further for hadoop-0.23
  • [BIGTOP-438] - need to bump the version of hadoop .23 to hadoop-0.23.3-SNAPSHOT
  • [BIGTOP-443] - deb/oozie/oozie-client.postinst installs an alternative for a path that isn't there
  • [BIGTOP-448] - Cannot install flume-ng-node pkg
  • [BIGTOP-449] - hadoop package cleanup for hadoop-0.23
  • [BIGTOP-450] - repackage Oozie as a true webapp
  • [BIGTOP-453] - hadoop needs to depend on Zookeeper
  • [BIGTOP-455] - we still need at HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec to be present in our launcher scrtips
  • [BIGTOP-461] - hadoop-httpfs unconditionally removes the alternative
  • [BIGTOP-462] - JSVC_HOME location can be both /usr/lib/bigtop-utils and /usr/libexec/bigtop-utils
  • [BIGTOP-464] - There's currently a commented block in puppet/modules/hadoop/templates/yarn-site.xml that fails to render in some cases
  • [BIGTOP-467] - [puppet] yarn historyserver currently doesn't depend on the mapreduce keytab as it should
  • [BIGTOP-468] - [puppet] Fix httpfs dependencies and give it its own keytab
  • [BIGTOP-469] - [puppet] web authentication principal missing from hdfs-site.xml
  • [BIGTOP-472] - make deb fails when building Whirr - OutOfMemoryError: PermGen space
  • [BIGTOP-473] - zookeeper man page needs to be updated
  • [BIGTOP-474] - improve Oozie integration with Hadoop 0.23
  • [BIGTOP-477] - launcher script for pig, hive, sqoop and mahout should not source /etc/default/hadoop anymore
  • [BIGTOP-478] - perhaps we should go back to mapred as a user name for mapreduce daemons in Hadoop 0.23
  • [BIGTOP-479] - init.d scripts should provide an option for initializing persistent state of the services that require it
  • [BIGTOP-480] - log aggregation knob is now yarn.log-aggregation-enable and needs to be enabled
  • [BIGTOP-481] - fix the way to handle repo key for Debian like system
  • [BIGTOP-482] - HTTPFS puppet code needs to be updated
  • [BIGTOP-484] - Fix typo error ('alredy' -> 'already')
  • [BIGTOP-485] - transition package dependencies to hadoop-client package
  • [BIGTOP-487] - oozie security configuration needs to be updated
  • [BIGTOP-488] - TestCLI missing dependency
  • [BIGTOP-493] - [puppet] core-site.xml and httpfs-site.xml end up changing every few times puppet is run
  • [BIGTOP-496] - Hive ships some source code in the package.
  • [BIGTOP-498] - Oozie's home dir is currently set to /var/run/oozie but should be /var/lib/oozie
  • [BIGTOP-500] - Oozie hive workflows are broken because of the HADOOP_HOME being deprecated
  • [BIGTOP-501] - it would help to have libext point to /var/lib/oozie in Oozie packaging
  • [BIGTOP-502] - Flume NG init script should pass --name "agent" instead of --name "<hostname>"
  • [BIGTOP-503] - remove unsupported upstream launcher scripts from packaging
  • [BIGTOP-505] - Flume should exit 0 when trying to stop a non-running service
  • [BIGTOP-506] - Flume should exit 0 when trying to start an already running service
  • [BIGTOP-507] - sqoop deb package is empty
  • [BIGTOP-509] - make all is failing because flume-1.0.0-incubating.tar.gz does not exist in APACHE_MIRROR
  • [BIGTOP-510] - log4j property must be set with property in for Flume NG pkg
  • [BIGTOP-512] - oozie can't be installed on debian
  • [BIGTOP-513] - Pig Packaging Cleanup
  • [BIGTOP-516] - it would be cleaner to get rid of mapreduce entries in limits.d/mapreduce.conf now that we consolidated the user back to mapred
  • [BIGTOP-519] - /etc/default files are not marked as configs by various hadoop packages
  • [BIGTOP-521] - Hadoop Pseudo-conf files have wrong permissions
  • [BIGTOP-526] - it could be useful to have dfs.permissions.enabled set to true in our configs
  • [BIGTOP-528] - A small patch to give better return codes which can be understood by chef and pupper among others.
  • [BIGTOP-529] - [puppet] We don't currently format zookeeper nodes on first run
  • [BIGTOP-530] - [puppet] We currently xst the HTTP principal multiple times, each time invalidating the previous one
  • [BIGTOP-537] - provide a workaround for non-LSB compliant catalina.sh handling of a stop action
  • [BIGTOP-540] - hadoop package on debian is missing JSVC_HOME specification
  • [BIGTOP-541] - Build issue for hadoop RPM
  • [BIGTOP-543] - fuse_dfs_wrapper.sh should not be provided
  • [BIGTOP-544] - zookeeper-server init doesn't work
  • [BIGTOP-546] - Hadoop rpm is missing fuse default file
  • [BIGTOP-548] - we should consider moving the creation of the hive user into the hive package
  • [BIGTOP-552] - Enable HDFS durable sync
  • [BIGTOP-554] - rat is complaining again
  • [BIGTOP-556] - hbase, yarn and hdfs users are getting incorrect home directories on Debian
  • [BIGTOP-557] - /etc/oozie/conf and oozie-conf alternative should belong to oozie instead of oozie-client
  • [BIGTOP-558] - debian description can't have the following set of characters ${VAR}
  • [BIGTOP-559] - HttpFS fails to load Hadoop configuration after installation.
  • [BIGTOP-560] - Flume-ng does not depend on ant or xml-commons anymore
  • [BIGTOP-561] - HBase packaging has dangling symlinks
  • [BIGTOP-562] - fix Mahout smoke test in trunk branch
  • [BIGTOP-563] - Build behind proxy fails
  • [BIGTOP-564] - Build failure for big-top trunk for pig, oozie, and whirr
  • [BIGTOP-565] - catalina-based services should be really careful with catalina stop
  • [BIGTOP-568] - Move the configuration directory for hadoop to /etc/hadoop
  • [BIGTOP-574] - Add netcat and fuser as dependencies for some of the Bigtop stack.
  • [BIGTOP-579] - hadoop-yarn creates log files with inconsistent ownership on Debian/Ubuntu
  • [BIGTOP-580] - a wrapper for the ZooKeeper cleanup script is missing
  • [BIGTOP-582] - when running Hadoop daemons pidofproc is mistakenly looking for 'su' instead of 'java'
  • [BIGTOP-583] - "Class path contains multiple SLF4J bindings" warnings
  • [BIGTOP-585] - hbase package has dangling symlinks
  • [BIGTOP-586] - HBase package should provide a versionless symlink to a jar for other projects to depend upon
  • [BIGTOP-587] - Hive package needs to depend on Zookeper and HBase
  • [BIGTOP-588] - Oozie puppet code needs to take care of initializing the DB
  • [BIGTOP-589] - TestLoadAndVerify's HBase scanner caching setting too high
  • [BIGTOP-590] - hadoop man page needs to be updated
  • [BIGTOP-592] - clean up content of conf.empty and conf.pseudo config files
  • [BIGTOP-593] - improve the package test to handle [arch=xx] in the repo file
  • [BIGTOP-595] - flume-ng agent should honor /etc/default settings
  • [BIGTOP-596] - move service configuration from groovy code to package xml manifest file
  • [BIGTOP-597] - Flume init scripts should not create the log and lock directories.
  • [BIGTOP-598] - missing zookeeper init on debian
  • [BIGTOP-599] - /var/lock/subsys may not exist on some distributions and this create some issues in Apache Hadoop
  • [BIGTOP-600] - hbase init script should not source hbase-config
  • [BIGTOP-607] - remove python build dependency from Hive package
  • [BIGTOP-608] - remove source/build artifacts from the Pig package
  • [BIGTOP-609] - add support for openjdk 1.7
  • [BIGTOP-616] - Fix TestHadoopSmoke and TestHadoopExamples: tests look for streaming and examples jar in wrong directory
  • [BIGTOP-626] - small misc. issues with the newly introduced Giraph package
  • [BIGTOP-630] - Fix services priorities
  • [BIGTOP-633] - bigtop-utils packages should depend on 'which'
  • [BIGTOP-636] - a few minor tweaks to the Hue deployment code
  • [BIGTOP-638] - move <services> block from common package_date.xml to each platform's (yum, apt, zypper) package_data.xml
  • [BIGTOP-639] - sqoop-metastore cannot stop because "Could not load db driver class: org.hsqldb.jdbcDriver"
  • [BIGTOP-642] - Suse does not have a package named which
  • [BIGTOP-643] - Fix package statement in TestHDFSQuota
  • [BIGTOP-646] - update metadata for Hue in the package testing manifest
  • [BIGTOP-647] - service flume-node fail to start
  • [BIGTOP-648] - hbase-thrift cannot be started properly
  • [BIGTOP-649] - PackageTestCommon.checkRemoval() is broken
  • [BIGTOP-650] - Override libexecdir for bigtop-utils on mageia

Improvement

  • [BIGTOP-266] - Need to override default yarn task classpath
  • [BIGTOP-273] - Need to update puppet recipies for Hadoop 0.23 deployment
  • [BIGTOP-313] - Move puppet configuration out of manifest and into extlookup() CSV
  • [BIGTOP-322] - update .gitignore to ignore backup files and eclipse generated files.
  • [BIGTOP-332] - it would be nice to bump hadoop version to 0.23.1-SNAPSHOT across the stack
  • [BIGTOP-366] - Set some minimum versions for ZK + Flume
  • [BIGTOP-371] - Update bigtop repository url for the applance
  • [BIGTOP-454] - update oozie man page
  • [BIGTOP-465] - webhdfs isn't in the puppeted hadoop configuration
  • [BIGTOP-466] - Secure zookeeper support missing from puppet
  • [BIGTOP-470] - [puppet] Improve secure configuration for zk and hbase
  • [BIGTOP-475] - Provides initscript for the REST HBase gateway
  • [BIGTOP-522] - Minor Hive Packaging Cleanup
  • [BIGTOP-534] - Need init.d script for zkfc
  • [BIGTOP-535] - add httpfs config property pointing to hadoop conf dir
  • [BIGTOP-555] - Bump version of HBase to 0.92.1
  • [BIGTOP-567] - Make sure we can detect the JAVA_HOME for java-package built JDK package
  • [BIGTOP-572] - Fix HDFS directory values in pseudo-conf hdfs-site.xml for 0.23+ NN/DN/SNN
  • [BIGTOP-573] - we no longer have to override container-executor.conf.dir
  • [BIGTOP-577] - [base conf] There's no hadoop-metrics2.properties present in the configs that are shipped.

New Feature

Task

Test

  • [BIGTOP-321] - Make TestLoadAndVerify parameterizable and runnable from command line
  • [BIGTOP-444] - Add sqoop smoke tests
  • [BIGTOP-569] - fix for TestCLI failures due to change in behavior of hadoop fs -put
  • [BIGTOP-610] - Add basic hdfs fsck test and move logError helper to bigtop-test-framework
  • [BIGTOP-611] - Add HBase TestImportTsv
  • [BIGTOP-618] - Add HBase hbck test
  • [BIGTOP-619] - Add HBase CopyTable test
  • [BIGTOP-620] - Add test for dfsadmin commands
  • [BIGTOP-622] - Add test for HDFS Balancer
  • [BIGTOP-623] - Add test for HDFS Quota Testing
  • [BIGTOP-627] - Add HBase balancer test
  • [BIGTOP-631] - Add HBase Import/Export test
  • [BIGTOP-641] - Add hdfs fsck options in TestFsck

Release Notes - Bigtop - Version 0.2.0

Sub-task

  • [BIGTOP-54] - Bump version of oozie to 2.3.2
  • [BIGTOP-55] - Bump version of sqoop to sqoop-1.3.0
  • [BIGTOP-56] - Bump version of hive to hive-0.7.1
  • [BIGTOP-80] - Add package dependencies to Zookeeper RPM for programs invoked from install/uninstall scripts
  • [BIGTOP-120] - Bump version of pig to 0.9.1
  • [BIGTOP-121] - Bump version of HBase to 0.90.4
  • [BIGTOP-167] - All of Hadoop wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [BIGTOP-171] - All of Pig wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [BIGTOP-175] - All of HBase wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [BIGTOP-178] - All of Flume wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [BIGTOP-179] - All of Hive wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [BIGTOP-180] - All of Mahout wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [BIGTOP-182] - All of Oozie wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [BIGTOP-184] - All of Sqoop wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [BIGTOP-188] - All of Whirr wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [BIGTOP-190] - All of Zookeeper wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils

Bug

  • [BIGTOP-3] - gitignore files need to be removed from SVN
  • [BIGTOP-4] - Remove references to Cloudera's internal infrastructure
  • [BIGTOP-15] - hadoop ubuntu packages need to have libssl-dev as a build dependency
  • [BIGTOP-20] - hard-coded pid location in Hadoop init.d scripts
  • [BIGTOP-25] - Standardize JAVA_HOME detection
  • [BIGTOP-27] - Add disclaimer text
  • [BIGTOP-35] - /usr/share/doc/flume and /etc/flume are owned by flume:flume
  • [BIGTOP-37] - The following config files are not marked as %config on RPM systems
  • [BIGTOP-40] - iTest package names/groupIds need to be moved to org.apache rather than com.cloudera
  • [BIGTOP-43] - Whirr and HBase aren't actually building from source in Debian packages
  • [BIGTOP-45] - Fix build issue for Mageia and openSUSE
  • [BIGTOP-46] - hadoop-native Debian package lists libz1 as Depends, libz-dev as Build-Depends
  • [BIGTOP-47] - Invalid download URL for whirr
  • [BIGTOP-48] - Fix pig for Mageia
  • [BIGTOP-50] - Flume init script's dependency on syslog needs to be removed
  • [BIGTOP-57] - Apt repo creation fails due to invalid distributions file
  • [BIGTOP-61] - HBase fails to add HADOOP_CONF_DIR to its classpath
  • [BIGTOP-62] - /usr/lib/hbase and everything within is owned by hbase:hbase
  • [BIGTOP-63] - Hbase leaves some unnecessary directories and symlinks when being uninstalled
  • [BIGTOP-66] - Packages need "Bigtop" identifier in version string
  • [BIGTOP-68] - bigtop-tests/test-artifacts have all their code in src/test, but those aren't tests to be run in that context
  • [BIGTOP-69] - Certain tests in iTest common require extraneous dependencies and should be @Ignored for now
  • [BIGTOP-71] - Unit test failures in bigtop-test-framework due to multiple Ant versions on classpath
  • [BIGTOP-72] - Gmaven plugin needs to be explicitly bound to compilation goals
  • [BIGTOP-73] - oozie-client can't be safely removed on Debian
  • [BIGTOP-74] - Add Mahout to package readiness tests
  • [BIGTOP-75] - Mahout package is broken
  • [BIGTOP-77] - All of our config files should be marked %config and (noreplace)
  • [BIGTOP-81] - oozie-setup.sh script fails if Oozie has not been started once
  • [BIGTOP-83] - Advanced checks in checkRemoval need to be moved out
  • [BIGTOP-84] - Package removal fails in case alternatives are deleted or damaged
  • [BIGTOP-85] - hadoop package ships native files
  • [BIGTOP-86] - Misc. issues with Oozie package
  • [BIGTOP-87] - The following packages don't remove alternatives: flume, hadoop-pig, hadoop-hive, hadoop-hbase
  • [BIGTOP-88] - Mahout package contains actual Hadoop jar, rather than symlinking to it
  • [BIGTOP-91] - Docs should be in %{_docdir}/<PACKAGE_NAME>-<PACKAGE_VERSION>
  • [BIGTOP-92] - Flume's dependency on ZK needs to be the same for DEB as it is for RPM
  • [BIGTOP-93] - ZK dependencies can not be satisfied on SUSE
  • [BIGTOP-94] - chkconfig startup priorities need to be normalized
  • [BIGTOP-97] - default files need to be normalized and moved to common
  • [BIGTOP-99] - Fix Sqoop for Mageia
  • [BIGTOP-102] - daemons need to be taken care of when packages are installed/upgraded/uninstalled
  • [BIGTOP-104] - hadoop daemons can not be started because hadoop-config.sh resets HADOOP_HOME
  • [BIGTOP-108] - oozie docs are installed at the incorrect location
  • [BIGTOP-109] - Hadoop should not format the namenode on installation
  • [BIGTOP-110] - DN can't be started in secure configuration on SLES
  • [BIGTOP-113] - redhat-lsb is obviously not available on Mageia
  • [BIGTOP-114] - Pig RPM package contains more files than it should
  • [BIGTOP-115] - Pig do-component-build hardcodes dist.dir as debian/tmp
  • [BIGTOP-119] - Create version-less symlinks of hadoop jars
  • [BIGTOP-122] - enable hadoop-sbin package
  • [BIGTOP-123] - enable hadoop-fuse package
  • [BIGTOP-124] - documentation on SLES is not supposed to have package version embedded in the directory name
  • [BIGTOP-125] - whir package is full of build artifacts
  • [BIGTOP-126] - zookeeper rpm should require groupadd/useradd
  • [BIGTOP-127] - BIGTOP-120 broke build on at least Fedora (and Mageia)
  • [BIGTOP-128] - BIGTOP-90 broke hadoop on Mageia
  • [BIGTOP-129] - oozie init.d scripts should transition to using oozied.sh from upstream
  • [BIGTOP-130] - Mahout package has an unneeded dependency on chkconfig
  • [BIGTOP-131] - RPM dependencies for flume need to be optimized
  • [BIGTOP-132] - hadoop datanode on SLES returns an incorrect code on stop/start
  • [BIGTOP-133] - hadoop packages should set up nofile limits automatically
  • [BIGTOP-134] - hbase packages should set up nofile limits automatically
  • [BIGTOP-136] - hadoop package still leaves binary files in /usr/lib/hadoop/lib
  • [BIGTOP-137] - task-controller needs to be rebuilt
  • [BIGTOP-138] - task-controller binary needs to be root:mapred with 4750 permissions
  • [BIGTOP-139] - links to jsvc and task-controller need to be provided in order to make hard-coded values work
  • [BIGTOP-140] - need to workaround HDFS-1943
  • [BIGTOP-141] - hadoop now requires an explicit declaration of a HADOOP_SECURE_DN_USER in order to run a secure datanode
  • [BIGTOP-142] - need to update zookeeper manifest to the state of packages
  • [BIGTOP-143] - need to update hadoop manifest to the state of packages
  • [BIGTOP-144] - kerberos deployment needs to be made compatible with default JRE crypto setup
  • [BIGTOP-145] - make cluster.pp be able to handle secure and unsecure deployments
  • [BIGTOP-148] - Hadoop fuse man page does not have the right permission and RPMs should not specify its extension
  • [BIGTOP-149] - HBase now requires commons-configuration jar to be in its classpath
  • [BIGTOP-150] - hadoop puppet deployment needs to install hadoop-sbin on datanodes when security is turned on
  • [BIGTOP-151] - need to provide more test coverage for HBase testing
  • [BIGTOP-152] - TestHadoopExamples needs refactoring into a parameterized test
  • [BIGTOP-154] - all of our /usr/bin wrapper scripts and init.d scripts need to start include find-java code from bigtop-utils
  • [BIGTOP-155] - need to implement a uniform way of versioning dependencies for test artifacts
  • [BIGTOP-156] - We should only run autoreconf when there is a compatible autotools installed
  • [BIGTOP-158] - oozie build pulls into oozie.war incorrect version of hadoop-core jar file
  • [BIGTOP-159] - oozie init.d script is not executable
  • [BIGTOP-160] - hadoop has deprecated HADOOP_HOME
  • [BIGTOP-161] - the new layout that Pig script adopted needs to be worked around
  • [BIGTOP-163] - stock hadoop doesn't support wildcards in proxy users
  • [BIGTOP-164] - lzo codecs need to be disabled everywhere in Bigtop
  • [BIGTOP-165] - oozie manifest includes functionality (sqoop and hive actions) that is not in oozie
  • [BIGTOP-166] - docs are missing from the mahout package
  • [BIGTOP-169] - Hadoop should buildrequire openssl-devel for centos builds
  • [BIGTOP-170] - bigtop-utils does not look for openJDK
  • [BIGTOP-173] - we still have references to hadoop-0.20 in our puppet code
  • [BIGTOP-174] - Does pig really requires git to build?
  • [BIGTOP-176] - Packages should set their mainteners to bigtop dev mailing list
  • [BIGTOP-181] - puppeted hadoop doesn't place PID files into /var/run/hadoop
  • [BIGTOP-183] - hadoop fuse manpage set to 755 instead of 644
  • [BIGTOP-185] - Running 'sudo /sbin/start flume-master start' on SLES11 doesn't have $USER variable.
  • [BIGTOP-189] - hadoop daemons do not run under the dedicated users (hdfs, mapred)
  • [BIGTOP-192] - oozie package needs to depend on unzip
  • [BIGTOP-193] - zookeeper doesn't generate any logs
  • [BIGTOP-194] - source is not a sh(1) command
  • [BIGTOP-196] - point Bigtop at the correct version of HBase inside
  • [BIGTOP-197] - Package testing artifact needs to be refactored
  • [BIGTOP-198] - hadoop-daemon.sh owned by many packages
  • [BIGTOP-199] - whirr docs are installed in the incorrect location
  • [BIGTOP-201] - bigtop package has an incorrect dependency on libhdfs.so
  • [BIGTOP-202] - zookeeper package is missing zoo.cfg
  • [BIGTOP-203] - sqoop needs to have access to hadoop .jars in its lib
  • [BIGTOP-205] - zookeeper-server doesn't do nohup before launching a damon
  • [BIGTOP-206] - during deployment time hdfs/mapred dir creation should be managed by hadoop module
  • [BIGTOP-207] - zookeeper doesn't allow for overriding the location of the PID file
  • [BIGTOP-208] - zookeeper does not depend on git or subversion
  • [BIGTOP-211] - Remove packaging code that will transition hadoop user/group to hdfs/mapred
  • [BIGTOP-212] - need to disable tests for features missing from Bigtop 0.2.0
  • [BIGTOP-214] - sequencing in puppet module for Hadoop needs to be imporved
  • [BIGTOP-215] - make site.pp a bit more friendly to how we start up cluster in Bigtop jenkins
  • [BIGTOP-216] - make oozie tests more robust in finding oozie-examples.tar.gz
  • [BIGTOP-217] - workaround Hive SQL parser in headers for our tests

Improvement

  • [BIGTOP-11] - Add Mahout to Bigtop
  • [BIGTOP-100] - Abstract component build logic from deb/rpm into common script
  • [BIGTOP-112] - Remove or replace textutils by coreutils dependency
  • [BIGTOP-195] - Document openSUSE 11.4 dependencies

New Feature

  • [BIGTOP-28] - Need to bump upstream version dependencies for the next release of Bigtop for hive, flume and sqoop
  • [BIGTOP-29] - Remove java dependency from all os types
  • [BIGTOP-30] - flume-master and flume-node packages should not own /usr/lib/flume/bin/flume-daemon.sh
  • [BIGTOP-31] - remove /usr/lib/whirr/cli/whirr.log from packaging
  • [BIGTOP-32] - Hadoop fuse package should include a file with defaults
  • [BIGTOP-34] - Add Hive server package
  • [BIGTOP-44] - Add a FAQ
  • [BIGTOP-95] - Puppet-based deployment infrastructure needs to be added to bigtop
  • [BIGTOP-111] - Add support for Mageia in the test framework

Task

  • [BIGTOP-59] - Reorganize source repo in a more self-explanatory manner

Release Notes - Bigtop - Version 0.1.0

Bug

  • [BIGTOP-2] - License headers need to be added
  • [BIGTOP-7] - Some components are pulling tarballs from archive.cloudera.com/tarballs rather than Apache
  • [BIGTOP-8] - Debian copyright files need to be updated for ASF in the right format
  • [BIGTOP-10] - Fix NOTICE file
  • [BIGTOP-13] - BIGTOP-7 introduced a version mismatch for Hive
  • [BIGTOP-18] - OrderedParameterized.java is not ASL-compatbile
  • [BIGTOP-23] - Out of heap errors when building pig
  • [BIGTOP-24] - Errors downloading tarballs from dist.apache.org via curl

Task

  • [BIGTOP-5] - Remove Cloudera references in packaging and test source.
  • [BIGTOP-6] - Add tarball target to top-level makefile
  • [BIGTOP-9] - Set up Bigtop podling website
  • [BIGTOP-14] - Add CHANGES.txt generation to Makefile