Support dependencies between optional components/extensions

Motivation
The RLS can now be installed as an optional extension via rustup. However, for it to function correctly, it requires the rust-src and rust-analysis extensions to also be installed. Ideally the user could just write rustup component add rls and the dependencies would be taken care of automatically.
Changes to manifest format
Here is a (very cut down) view of what a current manifest might look like in JSON:
(Click to expand)
{
  "date": "2017-04-27",
  "manifest-version": "2",
  "pkg": {
    "rust": {
      "version": "1.17.0 (56124baa9 2017-04-24)",
      "target": {
        "aarch64-unknown-linux-gnu": {
          "available": true,
          "hash": "b437bad2396b91649622340f2089da031d0d51a1c661915806f9e538f5666fc0",
          "url": "https://static.rust-lang.org/dist/2017-04-27/rust-1.17.0-aarch64-unknown-linux-gnu.tar.gz",
          "components": [{
            "pkg": "rust-docs",
            "target": "aarch64-unknown-linux-gnu"
          }],
          "extensions": [{
            "pkg": "rust-src",
            "target": "*"
          }]
        },
        "arm-unknown-linux-gnueabi": {
          "available": true,
          "hash": "b3254c556b3f427724af1f3a112e5ad93cffd61561b0690f54199ca465901a8f",
          "url": "https://static.rust-lang.org/dist/2017-04-27/rust-1.17.0-arm-unknown-linux-gnueabi.tar.gz",
          "components": [{
            "pkg": "rust-docs",
            "target": "arm-unknown-linux-gnueabi"
          }],
          "extensions": [{
            "pkg": "rust-src",
            "target": "*"
          }]
        }
      }
    },
    "rust-docs": {
      "version": "1.17.0 (56124baa9 2017-04-24)",
      "target": {
        "aarch64-unknown-linux-gnu": {
          "available": true,
          "hash": "df345f87803c7e940453f56f3824244454238ac701014c10dbf6ed51b7417545",
          "url": "https://static.rust-lang.org/dist/2017-04-27/rust-docs-1.17.0-aarch64-unknown-linux-gnu.tar.gz"
        },
        "arm-unknown-linux-gnueabi": {
          "available": true,
          "hash": "a775d37f10d168d77f4dacad7284c8afc436c746b1a0ee1cab919a02afc59a24",
          "url": "https://static.rust-lang.org/dist/2017-04-27/rust-docs-1.17.0-arm-unknown-linux-gnueabi.tar.gz"
        }
      }
    },
    "rust-src": {
      "version": "1.17.0 (56124baa9 2017-04-24)",
      "target": {
        "*": {
          "available": true,
          "hash": "330ab4ad23e0cb0148756f878f560d91474d6b4830d748fad6ef72ba777fa648",
          "url": "https://static.rust-lang.org/dist/2017-04-27/rust-src-1.17.0.tar.gz"
        }
      }
    }
  }
}

To track dependencies, each extension or optional component will gain an optional "dependencies" key, which contains an array of objects, defining the name and target of the dependency:
(Click to expand)
        "aarch64-unknown-linux-gnu": {
          "available": true,
          "hash": "b437bad2396b91649622340f2089da031d0d51a1c661915806f9e538f5666fc0",
          "url": "https://static.rust-lang.org/dist/2017-04-27/rust-1.17.0-aarch64-unknown-linux-gnu.tar.gz",
          "components": [{
            "pkg": "rust-docs",
            "target": "aarch64-unknown-linux-gnu"
          }],
          "extensions": [{
            "pkg": "rust-src",
            "target": "*"
          }, {
            "pkg": "rls",
            "target": "aarch64-unknown-linux-gnu",
            "dependencies": [{
              "pkg": "rust-src",
              "target": "*"
            }]
          }]
        },

Changes to rust's build system
Rust's build system will need to understand which components depend on each other, and will need to encode those dependencies into the manifests it produces.
Changes to rustup
Rustup will need to read the dependencies from the manifest, and handle the following cases:
Action: A user installs a component with dependencies.
Expected result: The dependencies are automatically installed before installing the component, if not already installed
Action: A user uninstalls a component which is depended on by another
Expected result: User is prompted to uninstall both components, or cancel
Action: A user upgrades a toolchain which contains a component with new dependencies
Expected result: The new dependencies are automatically installed
Action: A user specifies several components to install, which may depend on each other
Expected result: The operation completes successfully, even though some of the components were "implied" by others.
Action: The user install a component with dependencies, or uninstalls a component on which another depends
Expected result: The CLI will indicate why these additional install/uninstall actions are happening.
cc @brson

This looks great @Diggsey. Thank you. I like that you showed all the rustup cases too. Those should be reflected in the test suite.

Action: A user uninstalls a component which is depended on by another
Expected result: User is prompted to uninstall both components, or cancel

I'm not sure how to handle this, considering that rustup is used for scripting. There are very few interactive rustup commands today, and they are fairly predictable (rustup-init is one). Today rustup component uninstall is non-interactive.
It may be more predictable for scripters here to either have rustup just remove the components that depend on the component being uninstalled, or to print an error, suggesting the user re-run with -y.

cc @nrc @jonathandturner

You know, I'd be fine with it saying "WARNING: component rls is still installed which depends on rust-src"


Support addWithReferenceLayers() in ini configuration file format

We need to support addWithReferenceLayers API in ini Configuration file format.
  /**
   * @brief add refering to reference layers.
   * @note This method does add the provided layers itself but adds a deep copy
   * of the passed layers to the model. The layers passed to this function can
   * be reused later.
   * @note @a reference is a set of layers connected each other to form a part
   * or whole graph which can be loaded to a model and can be run.
   * More specifically, the graph with a non cyclic, directed graph with all
   * node has either incoming or outgoing connection defined when considering
   * non-input layer is directly connected to the previous layer. 'input
   * layer' is defined as a layer that has no incoming connection and
   * (identified as @a start_layers or input shape is specified explicitly).
   *
   * @param reference a group of layers being referred to.
   * @param type type of reference layers
   * @param scope scope of added layers, identifier will be added
   * @param input_layers input layers which will be used to connect input layers
   * @param start_layers start layers which will be used to specify start of the
   * layers inside @a reference
   * @param end_layers end layers which will be used to specify end of the
   * layers inside @a reference
   * @param type_properties type dependent properties
   */
  virtual void addWithReferenceLayers(
    const std::vector<std::shared_ptr<Layer>> &reference,
    const std::string &scope, const std::vector<std::string> &input_layers,
    const std::vector<std::string> &start_layers,
    const std::vector<std::string> &end_layers, ReferenceLayersType type,
    const std::vector<std::string> &type_properties = {}) = 0;

Without this, we cannot run the time sequence model with ini file format.

:octocat: cibot: Thank you for posting issue #1941. The person in charge will reply soon.


ie8 button hover is incorrect.

Possibly also an issue in UIKit?

Related: https://github.com/govau/uikit/issues/312

This is an issue in govau/uikit and has been added as an issue.


Shared/common dynamic template data + personalizations

Issue Summary
As a developer, I want to be efficient with bandwidth and data that's being transmitted.
So when I need to send an email to 1,000 different people, and I use personalisations, it's great that I can send this in a single API request to Sendgrid.
However, there is still a lot of "waste" happening if I'm including dynamic template data which is the same for all the recipients
I am forced to include this in the dynamicTemplateData property of each individual personalisation.
It would be trivial to allow us to specify shared or common dynamicTemplateData in the root of the request data (as you would for a regular email without personalisations, along with possible specific dynamicTemplateData for each personalisation, and for Sendgrid to merge this on the server via a simple one level deep object merge, prior to sending emails.
This way, we won't have to repeat shared data 1000 times, and we'll be working together towards a faster and more efficient internet.
Example
The following example illustrates the problem. Now I know this is a trivial example, but imagine that the payload is significantly larger. For example, this email may be going out to 1000 attendees that signed up to an event, so we're passing in dynamic template data specific to each attendee (e.g. their name), but also shared dynamic template data pertaining to the information of the event they all signed up to, which could be a significant payload size.
Current API:
const data = {
  personalisations: [
    {
      to<EMAIL_ADDRESS>      dynamicTemplateData: {
        personal: 'A personal message for Person 1',
        shared: 'Common data which is the same for all recipients', //Repeats
      }
    },
    {
      to<EMAIL_ADDRESS>      dynamicTemplateData: {
        personal: 'A hidden message for Person 2',
        shared: 'Common data which is the same for all recipients', //Repeats
      }
    },
    {
      to<EMAIL_ADDRESS>      dynamicTemplateData: {
        personal: 'A secret message for Person 3',
        shared: 'Common data which is the same for all recipients', //Repeats
      }
    },
    //...x1000
  ]
}

Proposed API:
const data = {
  //A top-level dynamic data placeholder for shared/common dynamic data
  dynamicTemplateData: {
    shared: 'Common data which is the same for all recipients', //Only specified once!
  },
  personalisations: [
    {
      to<EMAIL_ADDRESS>      //Nested dynamic template data as is, for each individual personalisation
      dynamicTemplateData: {
        personal: 'A personal message for Person 1',
      }
    },
    {
      to<EMAIL_ADDRESS>      dynamicTemplateData: {
        personal: 'A hidden message for Person 2',
      }
    },
    {
      to<EMAIL_ADDRESS>      dynamicTemplateData: {
        personal: 'A secret message for Person 3',
      }
    },
    //...x1000
  ]
}

Later on the Sendgrid server (example code in JS):
//Merge common dynamic template data with personalisation specific ones
for (const item of data.personalisations) {
  item.dynamicTemplateData = Object.assign({}, data.dynamicTemplateData, item.dynamicTemplateData || {})
}

The proposed API could potentially save a lot of bandwidth and would be far more efficient, especially with large dynamic data payloads.
I am aware that this is not a NodeJS library specific issue, but as I work with this library exclusively I wanted to post it here and hope that this issue can be raised internally with the correct API team.
I also raised a similar issue last year (https://github.com/sendgrid/sendgrid-nodejs/issues/1154) which doesn't seem to have progressed much. Although note that this issue is slightly different, and I think far easier to implement at Sendgrid's end.

I do absolutely support this. I thought the API would behave like the proposed one but it doesn't.
It's a waste of bandwidth.

Hi @adamreisnz! Can you send an email to<EMAIL_ADDRESS>with this request? They will be able to generate a ticket for the correct internal Twilio Sendgrid team.

I will try, but I have reported quite a few bugs (in the Sendgrid user interface) to Sendgrid support already, and that has all landed on deaf ears, with the same bugs still present years later.

Interesting, my impression was that this was how the v3 api already worked :/ Source:

When sending an email with the v3 Mail Send endpoint, you define the various metadata about your message, such as the recipients, sender, subject, and send time, at the root level of a JSON request body. ...

If you set multiple recipients on the root node, they will all be listed in the e-mail.  That's something you usually don't want in business scenarios. Otherwise you'd basically expose your customers.


minor readbility improvement

just merge or statement, do not repeat

I guess you realized that such a condition doesn’t work as you would expect ;)

Coverage remained the same at 96.89% when pulling 00b4ff0e07ae2296a15c3337eee993599d978e8a on balkan:minor_readability_improvements into c578cbff8a931d8bec3c3e026c8908cb61bf71d4 on BertrandBordage:master.

Coverage remained the same at 96.89% when pulling 00b4ff0e07ae2296a15c3337eee993599d978e8a on balkan:minor_readability_improvements into c578cbff8a931d8bec3c3e026c8908cb61bf71d4 on BertrandBordage:master.

Coverage remained the same at 96.89% when pulling 00b4ff0e07ae2296a15c3337eee993599d978e8a on balkan:minor_readability_improvements into c578cbff8a931d8bec3c3e026c8908cb61bf71d4 on BertrandBordage:master.


snap

Added snapcraft.yaml to build a snap.

Description of the change:
Added snapcraft.yaml to build a snap package to install operator-sdk on Linux.
Motivation for the change:
Snap package's support installing applications across 41 Linux distributions.

@mikeroyal Thanks for the PR. I'm not too familiar with snap packages but in general I'm not sure we have a consensus on whether we're intending to maintain deb/rpm/snap pkgs for each release.
Mostly I don't see a reason not to just curl the SDK release binary directly from the release page.
@joelanford We should probably discuss if we want to maintain release packages going forward.
So far we do have a process to update the brew formula for the operator-sdk but that's pretty much sourcing the operator-sdk binary from each release.
https://github.com/Homebrew/homebrew-core/blob/master/Formula/operator-sdk.rb

Hi @hasbro17, I will say snaps are easier to maintain than deb/rpm because snaps can auto-update to new versions or auto-revert back to a previous version if something breaks on the system. I would also do my part in making sure the snap stayed up to date if there was a interest. So just think about it with other team members and see if there any interest.

I'm in the camp of maintaining a release binary and nothing more for now. We should not commit to a particular package manager without more discussion. There should be a proposal doc submitted first detailing which managers we'll maintain configs for and a discussion in that PR.

@mikeroyal Sorry, I forgot to follow up on this.
While snap does seem like it would be easier to maintain, our general consensus is that we would rather not put package manager specific code in the core SDK repo. While it may not be too much work to update now, it opens the door for maintaining more ways to install the SDK in the future.
Perhaps we can have add a snap repo for the SDK to https://github.com/snapcrafters
And hopefully if you're interested we can keep that updated after every release.

Hi @hasbro17,
Thanks for getting to back me. I think the idea of adding a snap repo for the SDK to https://github.com/snapcrafters is great :100: and I'll help out in anyway I can!


how to make it run with a source data in a .xlsx or a csv. file

Hi,
I have never worked with python. My objective is to make your code work with excel or csv file, as I have some collected twitter messages to be clustered. I have encountered two problems:

when i tried to run the example data, there is no output anywhere, the command line finished execution. Then nothing happened. Where can I see the output?
I don't know what exactly is the desired data source should look like, from code it looks like some html file. So how exactly can I parse a xlsx file with two columns: twitter id, twitter text?

Thanks!

Hi Vanessa, you do have python 2.7.x installed, right?
You want to run serial.py from the command line, like python serial.py eaxmple_data/Peerbelt.hello.bg.xs.chunked.zip. This will help you run the example.
Once that is working, you want to change the class PeerbeltLine to parse your csv. Given that you don't have html-formatted text, it is safe to remove the calls to BeautifulSoup.
Hope this helps.

Hi,
yes, I did install python 2.7.10. And run exactly as you recommended, but I didn't see any output in the console. Has it been output to somewhere else? Or what should I do additionally to check how the output look like?
Also, I didn't find the class "PeerbeltLine", do you mean this one "peer_belt_driver.py"?
Thanks!

Hi Vanessa,
You can use the peer_belt_driver.py code  or read_tweepy.py as examples on how to hook your code into a pipeline to process your tweets from the csv. You should need to write additional code to parse the csv / text file similar to what is is done in the peer_belt_driver.py to parse html files.
Hope this helps, please let us know if it doesn't.

Hello.
I seem to have the same problem.
I am trying to run the code in the in-memory database using your example data.
I have installed Python 2.7.10.
I have modified settings.py in order to select the in-memory database.
When I am trying to run serial.py from the command line as following
py -2 serial.py example_data/Peerbelt.hello.bg.xs.chunked.zip
although the process is completed without any errors, I can not find the output of that process.
Am I doing something wrong? Is a file created but I cannot find it? Or I am missing the point entirely?
Any help would be much appreciated

Hi Ilias Chanis,
This code was working on Google App Engine. The instructions on the home page should still work, though it's possible they have made changes in the platform that are not backward compatible.
If you find that's the case, I'm happy to help make it run in the "upgraded" environment. I suggest trying to run it as-is. And once that works, adapting it to the in-memory database.
Good luck, and if it fails in the AppEngine environment, I'm happy to get it working again in that environment.
Best,

Thank you for responding singhj.
So, if I understood correctly, there is no way to run the code entirely in your own PC, the only way is by creating a Google App?
Because as stated in your homepage, there is an option to run it using an in-memory database (which i thought it meant that you can run it entirely on your own machine).
Thank you in advance,


intrinsicContentSize

Why is the intrinsicContentSize, 10x10?

https://developer.apple.com/documentation/uikit/uiview/1622600-intrinsiccontentsize
By setting intrinsicContentSize, you allow the view to resize itself according to its internal components. As soon as you have the position constraints properly set, it doesn't matter the real value of intrinsicContentSize.


Keyboard covers text input area on mobile

This is a common problem.
I think that is a way to detect it. I remember reading about the fix a few weeks ago.
It would make this much more usable

I googled it and found about using ids although I haven't tested it and the jsbin doesn't work http://stackoverflow.com/questions/18110852/input-textbox-hidden-behind-keyboard-on-android-chrome
Anyway, this belongs to mdl itself more than vue-mdl and can be achieved on user land easily so there's no point in automatically doing so because not everyone wants the page to scroll automatically


Encourage callers to use $select and inform callers when there's default properties in the response

Is your feature request related to a problem? Please describe.
Today API consumers regularly call API endpoints without using $select. This impacts performance of client apps (and impacts service COGS too).  Additionally, APIs endpoints that return a default set of properties in a response don't inform callers that there are more properties available, and that they'll need to use $select - which is often a support call driver.
The API Gateway Service team is trying to address these related issues by including a new "tips" instance annotation in the response to alert developers and encourage them to use $select, if $select is not being used.  This annotation is also only added if the client application provides an opt-in "ms-graph-dev-mode" Prefer request header
Describe the solution you'd like
As one the the premium tools for API discoverability and testing, we'd like Graph Explorer to send the prefer: ms-graph-dev-mode request header, so that the tips instance annotation shows up.
Describe alternatives you've considered
NA - although we vetted a range of ideas with Identity Developer Advisors.
Additional context
Feature spec

This feature is now fully deployed, and available for anyone to use.  Please note that for workloads that adopt a passthrough strategy, AGS will not provide the @tips annotation - so this is expected when calling APIs like /messages, /events or personal contacts.

Hey @dkershaw10 ,
I have created a pull request with the fix, feel free to test this out.
Question: I noticed it only works for the instances where a user is signed in, is this by design?

Nice!!!  Not sure I'll be able to have time to test it.
Question: I noticed it only works for the instances where a user is signed in, is this by design?
Not sure how your experience works for the "non signed-in" case.  That shouldn't really have any bearing on the annotation.  Is the header still being sent for your non-signed-in case?

Nice!!! Not sure I'll be able to have time to test it.
Question: I noticed it only works for the instances where a user is signed in, is this by design? Not sure how your experience works for the "non signed-in" case. That shouldn't really have any bearing on the annotation. Is the header still being sent for your non-signed-in case?

In the non signed-in case, the tips annotation doesn't show up.
Yeah I'm sending the header but it doesn't have an effect on the responses


v21.2.10 optimise BattlefieldMock

[Current coverage][cc-pull] is 100%

Merging [#550][cc-pull] into [master][cc-base-branch] will not change coverage

@@           master   #550   diff @@
====================================
  Files          28     28
  Lines         439    439
  Methods       111    111
  Messages        0      0
  Branches        0      0
====================================
  Hits          439    439
  Misses          0      0
  Partials        0      0

Powered by Codecov. Last updated by [4877f2d...5c69428][cc-compare]
[cc-base-branch]: https://codecov.io/gh/eugene-matvejev/battleship-game-api/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/eugene-matvejev/battleship-game-api/compare/4877f2d386f3909e6f8d12628941dbd4f5036a66...5c694281d27756aeb3b2804484bc68e2ccd3ac65?src=pr
[cc-pull]: https://codecov.io/gh/eugene-matvejev/battleship-game-api/pull/550?src=pr


Create an official typescript and typescript-compiler packages

Create an official typescript and typescript-compiler packages and talk with and deprecate - https://github.com/meteor-typescript/meteor-typescript-compiler
Follow also the typescript label.
Also, merge https://github.com/meteor-typescript/meteor-typescript-libs into the typescript package

Also, merge https://github.com/meteor-typescript/meteor-typescript-libs into the typescript package

It's actually bad idea to install typings via any package.
 In any case, it's a waste of time keep updating them in the package whereas they are updated constantly by the authors or community in the global repos.
For other definitions not available globally, we could create a Github repo for each particular definitions and recommend to install them using typings tool, while we'll send PR to include them into the global typings registry. This  tool can install definitions from a Github repo, NPM package, or even HTTP path.

ok sounds good about the typings libs repo.
let's add @orefalo and @jasonparekh to the discussion.

we also want to add a chapter on Typescript in the Meteor guide

adding @fullflavedave
perfectly fine with the idea.

Not having much context on all this, it looks like this is part of an awesome effort that could greatly improve the experience of using TypeScript with Meteor.   Are you all creating a new build compiler package that will make using TypeScript with Meteor almost as easy as using CoffeeScript?
I'm happy to deprecate meteor-typescript-libs if a better solution makes sense.

beta version of the future typescript packages is published as<EMAIL_ADDRESS>Packages involved:

meteor-typescript NPM package https://github.com/barbatus/meteor-typescript
typescript-compiler package https://github.com/barbatus/typesctipt-compiler
typescript package itself https://github.com/barbatus/typescript

Hey, I just saw this repo due to your comment.
How are multiple tsconfig's handled? TypeScript 1.8 implemented the handling of tsconfig files closest in the path of a ts file. This is especially important for isomorphic code, because definition files, declaring i.e. a class "Plugin", are completely different things i.e. on the client vs. the server. See here a feature illustration that was implemented in Webstorm: https://youtrack.jetbrains.com/issue/WEB-17472#comment=27-1098038
Also, last year I investigated what is necessary to bring TypeScript to meteor (we stopped because of missing module support and currently use webpack), for which I wanted explicitly TypeScripts incremental build feature because TypeScript code bases can become rather large for which the compile method creates a lot of overhead. A good way is to use the native language service API. You can read up on it here: https://github.com/Microsoft/TypeScript/issues/4382#issuecomment-133188488

@D1no
While multiple configs feature is likely doable using current state of the Meteor compiler API , it’ll significantly overcomplicates things.
First off, we are trying to create/test some initial version, and then will update it gradually. Not to mention that tsconfig.json itself in different IDE has multiple additional options, for example, filesGlob is specific to WebStorm and Atom. At the same time, original tsconfig.json is quite simple and has only “files” options. I see the right strategy is to stick to TypeScript updates itself since, I believe, features, like mentioned filesGlob or even multiple configs, will be implemented in some way in TypeScript itself one day . We, of course, might implement them faster in case they were requested by multiple users and/or certainly makes sense for Meteor.
BTW, isomorphic ( in the Meteor context) i believe means when some code base has same API whether on the client or server if it’s supposed to work both on the client/serve. Otherwise, if APIs are different and can conflict in a definition file, I don’t see why methods/classes can't be renamed to contain, say, ‘client’/‘server’ identifications to outline differences and avoid conflicts.
About the last one. Yes, I plan to implement some in-memory files cache to avoid reading overhead.

@D1no I agree with @barbatus approach and hope to get a major release for that package soon without that feature.
But, if you like to collaborate together on a branch with this feature I would love to do that

Hey @barbatus, thanks for your reply! I tried to find it - but couldn't - that TypeScript (the language service api) already supports multiple tsconfig files by recursively running from a file down the folder structure picking the first tsconfig it finds. Identical behaviour to what is found in the webstorm ticket. But maybe I am wrong.
Also, I fully embrace that this is prototyping for a functional compiler and the TypeScript project is far from simple. I already did a tech evaluation a while back on it... of course it is not so present in my mind as it should be. But here some feedback to your reply - stuff you may know - which may leap up deep into further versions.
(Edit: I didn’t finish the comment last night. In retrospect, could have been a blog post... typescript in isomorphic environments is still a pain.)
Importance of tsconfig.json
The tsconifg.json has far more official options than just files. For meteor, the important one here is the flag --noLib that prevents the inclusion of lib.d.tsand provides the ability to define additional d.ts typings. So you need different tsconfig's to not only specify compile targets but also typing configurations. Why?
Beware, lib.d.ts is included by default
By default, the TypeScript compiler includes a lib.d.ts that tries to be magic by typing a whole bunch of global variables that you may find in the DOM; ES5, ES6 & ES7 Specs, Webworkers etc. If those global typings are included by default, you'll see that you get a bunch of errors when trying to use TypeScript on the Server because there are many Globals/Classes etc. typed through the standard lib, that work completely different (wrong shape) or collide with additional typings of i.e. node.d.ts or meteor.d.ts. The same applies to other libraries & compile targets.
Danger of double Typing
This is i.e. the case with Plugin. It is typed in dom.generated.d.ts which is part of lib.d.ts. But Plugin is also a global Class of Meteor - as you know (the build plugin).  That is also the reason why the meteor typescript libs have been split up into server, client, common etc (https://github.com/meteor-typescript/meteor-typescript-libs/issues/25).
Plugin is just one of many many examples where the same class/global/method names mean completely different things depending on the build target (client web / cordova / webworkers / etc. - server node / meteor / test runner).
Target dependent Typing
So in order to use TypeScript effectively, you really need to slim down the default typings and handle them manually. Prevent the inclusion of lib.d.ts with the --noLib compiler flag and re-include them depending on the section of the app while not breaking the sharing of “common” code. In a best case, while IDE's still understand what's going on (= they need to see tsconfigs to adjust the IDE level diagnostics).
I.e. the code on the server should not have DOM typings but include core.d.ts. On the client, typings of es7.d.ts are probably overhead. Same for node, meteor etc typings. Which exact libs needs to be investigated. Because what doesn't work (mentioned above): You can not "re-type" classes because d.ts do not overwrite each other.
Language Service API as a quick way to get all functionality
The language Service API is capable at managing all files in a project. It is intended for fast re-transpiles. As a reference TypeStrong (ts-loader for webpack) implemented the language service api, though it is not an IDE, because the simple compile/transpile method falls apart on multi-file bases. Or getting corrupted results due to missing diagnostics. Using the language service api can be a quick win because it includes everything like the management of typings, in-memory caching, diagnostics etc. etc. Saving one to re-build everything around the method.
A reference how to do that can be found here (as mentioned above): https://github.com/Microsoft/TypeScript/issues/4382#issuecomment-133188488
Technically, the vanilla Language Service API expects file system access. But as TypeStrong did with its getScriptSnapshot method (linked through the comment above) it is replacing the file-system with an adapter into webpacks file management. The same can most likely be done with meteor -> simply giving the L.S. API access to meteors “file structure” by bridging file access inside the compiler. Provide a tsconfig. Done.
So this is a proposal to bridge to the Language Service API instead of re-creating everything around the transpile method. This would also mean, that typescript is easier to update. But I have to agree, it is really not that easy to get a first grasp of the monolithic structure.
I loosely consult two large projects that use TypeScript through and through and after a long evaluation and a few failed attempts to bring TypeScript to meteor, webpack was chosen.
I lobby every now and then for the support of TypeScript (https://github.com/meteor/guide/issues/41#issuecomment-151774403). I also own the atmosphere.com/typescript Org (the experimental package has been pulled; but happy to include solid solutions that honer the ts team) and personally planned to revisit the issue in the summer. Though of course, atmosphere seems full of typescript packages now… and I am a little out of date on the issue

An additional FYI: TypeScript 1.8 is moving to the Salsa Engine which essentially allows you to completely mix TypeScript and JavaScript code. Thus, compiling ES6-7 JS code, incl. TypeScript and removing the need for the vanilla ecmascript package while improving type management.
https://github.com/Microsoft/TypeScript/issues/4789

@barbatus, I hope you don't mind me asking here: with the beta typescript package, what's the recommended way to handle references among an application's source files?  I'm assuming the goal of this package is (or perhaps proposing that it should be) to reach parity with CoffeeScript and JavaScript in terms of convenience for the developer.  I'm migrating an application from CoffeeScript, and I've run into some problems:

The TypeScript compiler is invoked on one file at a time, so I have to make all my files <reference> each other for the type information to be available.  For comparison, Visual Studio Code doesn't need this, assuming I have a tsconfig.json file.
Meteor adds a closure wrapper to each JavaScript file emitted by the compiler.  So if file A references a definition D in file B, even if the compiler thinks that D is in scope, the app will fail at runtime unless B copies D to the global object.  Given that the point of TypeScript is to catch as many errors as possible at compile time, it's disappointing to leave this class of errors to runtime.  Even worse, if multiple source files contribute to a TypeScript internal module of the same name, the compiler emits JavaScript code that is capable of merging the contributions into a single object if run in the global scope, but with the closure wrapper, each file generates its own partial object.  For the app to work at runtime, the developer would have to add code to copy all the definitions back and forth among these objects.

The hansoft:typescript package works around both problems by running the compiler once on all TypeScript files in the app and generating a single JavaScript file (which then gets the function wrapper): maybe not the best solution.  Another possible workaround to problem 2 is to disable the closure wrapping by setting the bare option on the JavaScript files.

@mattmccutchen Thanks that you tried it.
It’s not clear to me though what does “handle references among an application's source files” mean?
If one file uses a class from another you do “import {class} from ‘./another_file.ts’”.
TypeScript will resolve path to that another file, load it, then analyze its content, and finally complain if it doesn’t find that class.
<reference> that you mentioned is for typings essentially not for regular ts-files.
Though, recommended way to handle typings is via the tsconfig.json now instead of using <reference>s.
I didn’t get where is the issue specifically in the second part too.
If your file B has some class D then you either export it explicitly using ES6 modules (export class D {}) or make it global. If you make it global it doesn’t matter wrapped it in a function or not, you can use it everywhere. Yes, Meteor wraps each JavaScript file in its own function scope by default, but this is done for the purpose and has existed from the beginning. Otherwise all variable you define to be global in a file will become global to whole app (if not ES6 modules used), which is not Meteor devs are used to. I so far don't see why TypeScript should violate it.

@barbatus: I made an example app with two files and used the ES6 export and import syntax, but I get an error when I run it:
W20160225-03:44:06.392(-5)? (STDERR)
W20160225-03:44:06.394(-5)? (STDERR) /home/user/.meteor/packages/meteor-tool/.1.1.10.ki0ccv++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/server-lib/node_modules/fibers/future.js:245
W20160225-03:44:06.394(-5)? (STDERR) 						throw(ex);
W20160225-03:44:06.394(-5)? (STDERR) 						      ^
W20160225-03:44:06.394(-5)? (STDERR) ReferenceError: exports is not defined
W20160225-03:44:06.394(-5)? (STDERR)     at foo:1:17
W20160225-03:44:06.394(-5)? (STDERR)     at foo:3:2
W20160225-03:44:06.395(-5)? (STDERR)     at /home/user/objsheets/meteor-tutorial/typescript-test/.meteor/local/build/programs/server/boot.js:242:10
W20160225-03:44:06.395(-5)? (STDERR)     at Array.forEach (native)
W20160225-03:44:06.395(-5)? (STDERR)     at Function._.each._.forEach (/home/user/.meteor/packages/meteor-tool/.1.1.10.ki0ccv++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/server-lib/node_modules/underscore/underscore.js:79:11)
W20160225-03:44:06.395(-5)? (STDERR)     at /home/user/objsheets/meteor-tutorial/typescript-test/.meteor/local/build/programs/server/boot.js:137:5
=> Exited with code: 8

I must be missing something, perhaps a "module" setting in tsconfig.json.  Can you tell me what I'm missing for this app or give me another working example where one file refers to something defined in another?

ES6 exports (as well as most ES6 syntax) are not supported in the browsers currently. But there are a lot of polyfills libs for this particularly import/export feature including  CommonJS, SystemJS etc. So when you set "module:system" in the config, TypeScript will compile it to the SystemJS notation. But "compiler" doesn't mean it adds SystemJS lib itself to your project. You'll have to it yourself (to add this package https://atmospherejs.com/systemjs/systemjs).
By default, TypeScript package has  "module:commonjs" set internally, which means you can test it now too but with Meteor's modules package (https://atmospherejs.com/meteor/modules) added and run with Meteor 1.3 beta version.

@D1no Regarding multiple configs and “files” section, I meant that multiple configs might be implemented in the future, not that it’s already implemented. “files” section also means just “files” section nor any compiler options nor any other.
About Target dependent Typings. Yes, it makes sense. Thanks that you elaborated it now I see what you meant. Though, the case that you mentioned I believe can be handled without adding multiple configs nor any other external approach. Checking file architecture and extending API of the some internally used packages (for the current solution) should be enough to add/remove TypeScript specific typings where needed.
From other side, I’d say that CommonJS and NPMs are usually used for programming on the server side (which is NodeJS), which means there are not much libraries that define some global variables there (which, as you say, then might conflict with some client side API). Though, I am not saying its not possible (and never said). It just doesn’t seem to me that having multiple configs are crucial now.
We’ll let TypeScript package mature gradually while looking on what people want to see and issues they have.
About language Service API. First of all, current solution doesn’t seem to me as simple as "re-creating everything around the transpire method”. Internally the package uses TS program API, which can be quite extensively configured to take into account all Meteor environment’s advantages and specifics. It’s possible to intercept into files access, change module resolution and much more other etc.
For example, in this issue https://github.com/Microsoft/TypeScript/issues/4382#issuecomment-13318848 is in-memory files cache mentioned. It is now implemented in this package as well using an interception in the TS program host.
As I can see in the Language Service API docs, it says Language Service API is more like a long-lived  program. You seems to know much about Language Service API and grasped it already.
So question to you, what are real advantages bringing Language Service API to a Meteor TypeScript package (except potentially more architecturally brushed solution), if we know that Meteor watches files itself  for changes, it provides file contents, is able to cache results, and we can’t share file access between Meteor instances, will be? What does "fast re-transpiles" mean and how we can reach it in the Meteor context if makes sense? What advantages does it give for typings management (I didn't find info on this one)?

@barbatus, thanks for the info.  I was able to get the commonjs version to work, but I don't think I want to use a beta version of Meteor for my real app.  In the systemjs version, the app starts without error, but the code in typescript-test.ts is never executed (e.g., console.log statements do not print).  Am I missing a declaration that typescript-test.ts is the entry point to the application?  I imagine this is not a TypeScript-specific question and hate to burden you with it, but it doesn't seem to be documented (nothing useful here or here), so unless you point me to the answer, I won't be able to adopt the official TypeScript packages.

I answered my own question based on the SystemJS documentation: I need a separate TypeScript file that isn't a module itself, but just says System.import('typescript-test').  I've updated the example; hopefully it will be useful to the next person.  I'll consider this approach for the real app.

@barbatus: I want to add a little more about the way I originally tried to use TypeScript.  Our app is a research prototype, originally in CoffeeScript.  It quickly outgrew one file, but we were following pretty casual development practices and found it convenient to define globals in each file and refer to them in other files by unqualified names.  I felt (and still feel) that a per-file module system would be additional boilerplate for no practical benefit to us.  Over time, I wasted a lot of time tracking down stupid coding mistakes (passing the wrong number of arguments to a function, forgetting which class an object belongs to and accessing the wrong member, etc.), so I decided to move to TypeScript to catch such mistakes more quickly.
Code that defines properties on "this" to bypass Meteor's function wrapper works at runtime in JavaScript, CoffeeScript, and TypeScript, but cannot be type-checked by TypeScript (which is the whole point of using TypeScript); this is the way in which TypeScript differs from the other languages.  On the other hand, TypeScript is capable of treating all files as belonging to one scope (which can be further organized using TypeScript internal modules), and the TypeScript handbook encourages this mode for applications that aren't running in an environment that enforces the use of per-file modules (called "external modules" by TypeScript).  Within this mode, it's possible to emulate per-file and global scopes (like you'd have if each file had a function wrapper) in a manner compatible with TypeScript with a small amount of boilerplate by wrapping each file in an internal module of the same name.  Then you can control what's exported from each fragment of the internal module, and within each fragment, unqualified names resolve to items exported by other fragments.
I think there would be value in Meteor supporting the "all files in one scope" mode for TypeScript, given Meteor's emphasis on convenience.  As you saw, I was unaware of Meteor's support for external modules and assumed that this was the preferred approach.  In fact, I think it would be a more useful default than the current behavior for TypeScript files that are not external modules, but I would be happy even if were an opt-in.  But if you're set on promoting external modules as better software engineering practice, I'm not sure I can convince you.

@mattmccutchen The reason why using ES6 modules is a recommend approach now because Meteor's file processing does not guarantee often a file order you except. For example, even if you use global variables it's recommended to put everything with globals in a folder named “lib”. Files from this folder are guaranteed to be executed first allowing you to use these globals in other files confidently.
I see similar situation with TypeScript internal modules here. Why it works in TypeScript?
When you feed all files  in the tsc command, it is aware of reference syntax therefor it generates ultimately a bundle with the right file dependency order.
This won't be the case with Meteor. Meteor raises file change event on per file basis, so there is no way to compile all files at once every time generating one bundle. Otherwise, it will be a significant overhead.
I looked closely though into how TypeScript treats module {} construction. It turned out to wrap everything in the function scope exposing one global object with properties that were explicitly exported (marked export), which is kinda similar to global variables in Meteor. I am considering now adding bare option to files when TypeScript’s “module” set “none”. Let you know if I decide to do that. But again, splitting code into multiple related internal modules (like in that example with validators here http://www.typescriptlang.org/Handbook#modules-splitting-across-files) won’t be possible unless you guarantee the right file order, so you better off to use ES6 modules.

@barbatus: Good point about load order.  When I consider that, ES6 modules become closer to worthwhile for us if we had started to use them from the beginning.  This is nothing specific to TypeScript.  I assume that when the Meteor official support for ES6 modules comes out, they will be documented as recommended for all languages?  (I don't see that in<EMAIL_ADDRESS>yet.)
Thanks for the fix so far.  But note that, even when the module compiler option is not set to none, TypeScript allows you to have files that are not external modules (do not contain top-level import or export statements).  So the smarter approach would be to enable bare for each file that is not an external module, regardless of the module compiler option.  This would remove one possible stumbling block for new users (forgetting to set module to none).

When you feed all files in the tsc command, it is aware of reference syntax therefor it generates ultimately a bundle with the right file dependency order.
This won't be the case with Meteor. Meteor raises file change event on per file basis, so there is no way to compile all files at once every time generating one bundle. Otherwise, it will be a significant overhead.

I'm not so sure that compiling all files at once is slower.  For an extreme case, suppose you have 10 TypeScript files that follow a dependency order, with each file referencing all the files before it.  (You could even have cyclic dependencies, but I'm picking a less controversial example.)  Then when you modify the sixth file, you invalidate the compilations of the last 5 files.  Meteor will call tsc 5 times, and each time tsc will load between 6 and 10 files and then emit 1 file.  If Meteor compiled all files at once, it would call tsc once, and tsc would load 10 files and then emit 10 files.  The second way might be faster depending on the speed of emission compared to parsing.  But the way to really solve the problem is to use a long-lived Language Service like D1no said.  Then you tell the Language Service to reload the one source file that changed, and based on its internal data structures, it emits only the output files that need to change.
If you do compile all files together (either batch or using the Language Service), that will be great for users using globals because they won't have to write <reference> lines just to have type information available for globals in other source files.  They can still write <reference> lines to control the load order.

@mattmccutchen Modules are not just new Meteor’s feature, it’s part of ES6. Of course, nobody forces you (and won't) to use them, but common practice shows that they works much better for big projects (at least) than global variables approach or anything close.
This makes sense, thanks. Though, I need to check if there is a way in TypeScript API to get this kind of information for a file (to check if it exports anything).
No,  TypeScript program API is used currently in the implementation, not tsc itself.
So 5 calls to API for 5 files will likely be faster (if everything is cached internally) than calling tsc for 10 files. However I am now working on a version that will be based on Language Service API.

No, TypeScript program API is used currently in the implementation, not tsc itself.
So 5 calls to API for 5 files will likely be faster (if everything is cached internally) than calling tsc for 10 files.

You're right, the plugin uses the program API, but it creates a separate Program object for each file.  I'm pretty sure that Program objects are intended to be independent and no information is cached across them, and callers are supposed to use the language service if they want caching.  Do you have reason to believe otherwise?
As an aside, I ran into a problem trying to test different settings for the module option with the latest barbatus:typescript-compilers code (77e8a7c) and Meteor 1.3 (specifically, I used c1b57d6).  Because the build plugin in the published barbatus:typescript is bundled with its dependencies, I had to also use a local copy of barbatus:typescript (59b7e14).  I added a statement at the beginning of processFilesForTarget:
console.log('processFilesForTarget', inputFiles.map(f => this.getExtendedPath(f)));

And got this output:
processFilesForTarget [ 'packages/barbatus:typescript/runtime-tests.ts',
  'packages/barbatus:typescript/runtime-tests.tsx',
  'packages/barbatus:typescript/demo/tsconfig.json',
  'packages/barbatus:typescript/demo/client/demo.ts',
  'packages/barbatus:typescript/demo/typings/server.d.ts',
  'typings/browser/ambient/meteor/meteor.d.ts',
  'typings/main/ambient/meteor/meteor.d.ts',
  'typings/browser.d.ts',
  'foo.ts',
  'simple.ts',
  'tsconfig.json',
  'typings/main.d.ts' ]
processFilesForTarget [ 'packages/barbatus:typescript/runtime-tests.ts',
  'packages/barbatus:typescript/runtime-tests.tsx',
  'packages/barbatus:typescript/demo/tsconfig.json',
  'packages/barbatus:typescript/demo/server/demo.ts',
  'packages/barbatus:typescript/demo/typings/server.d.ts',
  'typings/browser/ambient/meteor/meteor.d.ts',
  'typings/main/ambient/meteor/meteor.d.ts',
  'typings/browser.d.ts',
  'foo.ts',
  'simple.ts',
  'tsconfig.json',
  'typings/main.d.ts' ]

Because packages/barbatus:typescript/demo/tsconfig.json comes before my application's tsconfig.json in the list, my tsconfig.json was completely ignored according to the logic in processConfig.  (With Meteor 1.2.1, the barbatus:typescript files were not in the list.)  I don't know if this will still happen once your packages are published and I use the published versions, but it seems there could be a broader problem if an app and a local package both have TypeScript code that is actually used and needs to be compiled with different options.

@mattmccutchen I guess, to be pretty sure about something one needs at least to check it out closely. I cache internally access to the files AST using the program host's interceptors. Program API is actually lower level API and used by the language service ultimately. Though, it's not all possible caching there.
This is terrible. I forgot that Meteor 1.3 processes now all files in the package folder (whether added by addFiles or not). Thanks for spotting it out (see this issue https://github.com/meteor/meteor/issues/6002). For now, I just removed that demo.

@mattmccutchen ok. What particular cases do you mean that may change the emitted code?
Meteor 1.3 is moving packages development to NPM, which means that Atmosphere packages will unlikely be popular afterwards. I don’t see any problem to add support for multiple configs (including packages) but it's not possible to have everything in the first version. Let’s see what features are really needed and move gradually. As usual, everybody will be able to add issues about features they want to see after package release (if it finally happens).

What particular cases do you mean that may change the emitted code?

main.ts:
///<reference path="import.ts"/>
module MyModule {
  console.log(x);
}

import.ts:
var x = 42;
module MyModule {
  //export var x = 43;
}

If you compile main.ts now, you get:
// main.ts
///<reference path="import.ts"/>
var MyModule;
(function (MyModule) {
    console.log(x);
})(MyModule || (MyModule = {}));

If you uncomment the export in import.ts and then recompile main.ts, you get:
// main.ts
///<reference path="import.ts"/>
var MyModule;
(function (MyModule) {
    console.log(MyModule.x);
})(MyModule || (MyModule = {}));

Notice the MyModule.x.

To recap, the remaining features I'd like for TypeScript apps that use globals or internal modules:

[ ] All files that are not external modules can see each other's declarations without using <reference>.  This is how tsc -p . and Visual Studio Code work, so I think it is what TypeScript users will expect as the default, even if compilation is slower.  You can always add an option to compile each file separately.  And external modules can always compile each file separately.  (It's possible for an external module to define globals or refer to globals defined in other files, but I'm satisfied to ignore those cases for now and let people patch them up with <reference>.)
[ ] All files that are not external modules are compiled as bare even when the module option is not set to none.

I filed Microsoft/TypeScript#7359 to make the ts.isExternalModule function public, but you can probably go ahead and start testing it.
Then there's the issue of the cache key, which I know you're still working on.  For internal modules, it can affect the emitted output; for external modules, AFAIK it only affects diagnostics, but I would still like the correct diagnostics.

@mattmccutchen

Emitting one js-bundle (for dev mode) is not a Meteor way to process source code since its API is not suited for that. This feature might appear one day but it won't happen soon. I'd say most people will use external modules anyways, so those who chose to use internal modules I don't see it as a problem if they need to use <reference> for now. Visual Studio Code may work great with TypeScript but we are actually adding TypeScript to Meteor, and as I see it, using TypeScript modules is much cleaner and less bugs-proven way comparing to using global variables (as it was before).

I just about to add it. ts.isExternalModule is not public? I think it's available now.

The case you mentioned when code needs recompilation seems to be a bit weird. Will see what we can do with it. For now, I am adding diagnostics re-examination for dependent files; it will be done soon.

Emitting one js-bundle (for dev mode) is not Meteor way to process source code since its API is not suited for that.

If you pass multiple files to the compiler without the outFile option, all the declarations will be available but the compiler will still emit individual output files.  Did you consider doing this?

ts.isExternalModule is not public? I think it's available now.

The function is accessible to third-party JavaScript code, but as of now, being "not public" means that the TypeScript maintainers reserve the right to break it and it's excluded from the public type declaration file typescript.d.ts to discourage third-party TypeScript code from calling it.

For now, I am adding diagnostics re-examination for dependent files; it will be done soon.

Great, let me know when this is ready for me to test.

Some quick updates on the last changes in the compiler and on the progress in general.
@D1no meteor-typescript compiler doesn’t make any difference between TypeScript declaration files and general TypeScript files, which means all TS files will be processed and
get into an architecturally-specific (elaborated further why) AST if they are placed in any folder within a Meteor app’s folder structure. At the same time, it makes difference which Meteor architecture each file to be processed has. So if a declaration file lies in the server folder, for example, it will be taken into account only for server TS files only, if it’s in the client folder - only for the client’s files, and lastly if it’s at the root level - it will be parsed for both architectures.
Regarding TypeScript default libraries, as I can see in the TypeScript repo now, they have refactored some default libraries in order to (as it seems to me) to make separation
between definitions of the Node and browsers environments clearer. For example, Plugin was moved to lib.dom.d.ts. While basic lib.d.ts still has some definitions specific to the browsers only but
they don’t seem to intersect with the Node and Meteor. If somebody be facing an issue with that in the future (which I doubt), we could create own copies of the default libs with even clearer separation - it’s not hard.
@mattmccutchen As to diagnostics re-evaluation when file references change, it should work now. Please verify that it works for you as well. One note though on rendering files barely in the automatic way when they don't export anything, i.e., what we're discussing some time ago. I was forced to remove this feature (content checking for exports), and leave only one way, i.e. when  module: none is set in the config, to turn on TypeScript internal modules. The reasons behind that is to make available imports like import ./foo for ES6 modules (i.e., when whole file is being imported). In other words, file foo.ts might not have exports but, at the same time, it still can be imported wholly using ES6 modules. If it’s rendered with bare set to true, it’s stripped of any modules information, therefore - no way to import ./foo. Since ES6 modules (CommonJS particularly) has more priority for Meteor 1.3, I decided to leave only module: none way for TypeScript internal modules.

Thanks for the updates @barbatus
I know this might be a little bit of an annoying question, but can you provide more context around @Urigo's first post in this thread and give a brief explanation of how the current meteortypescript:compiler relates to all of the other packages listed here, and what the future plan is.
I have spent the past half hour trying to answer a basic question: what project can I fork to bump the official typescript package version (tsc)?
However, I feel like I am going a bit mad tab-jumping to a bunch of different repositories, and its a little bit hard to get the "Big Picture" of Typescript in Meteor.

@jacobdr meteortypescript:compiler is another one TypeScript package which doesn't relate directly to the current effort going on in the this issue. I have though repo access to meteortypescript:compiler and plan in the near future to add (imply) https://github.com/barbatus/typescript package there in order to test it on a bigger auditory. After, if https://github.com/barbatus/typescript turns out to be ok for most users, I have plans to deprecate meteortypescript:compiler.

@barbatus I think @jacobdr makes a good point that we haven't documented well the goal and relashanships of all the packages that form the new Typescript package.
I think we should specify in their readme files what they do, how they related to each other and how does it relate to the ecmascript package model:

https://github.com/barbatus/typescript
https://github.com/barbatus/meteor-typescript
https://github.com/barbatus/typescript-runtime
https://github.com/barbatus/typescript-compiler

Sounds good. To be clear: as a Typescript consumer looking for the latest Meteor build compiler, I want to meteor add barbatus:typescript-compiler to my project, correct? (With the aim of deprecating or pointing meteortypescript:compiler to the new effort?)
I would be happy to contribute to the documentation/standardization of packages effort, but to do so I probably need some context on the topic. Is there something in the Angular-Meteor design docs or Trello that you guys might be able to point me to?

@jacobdr I've changed meteortypescript:compiler to use barbatus:typescript recently.
The source code to my mind is quite simple and easy to get into. I've also added some commentaries there recently, and plan to add more. So you could try to look into the source code and learn.
Package structure is briefly described here https://github.com/barbatus/typescript#package-structure.

@fullflavedave Can you please look/approve Definitelytyped PR?

I just approved it, but a few, easily fixed lines (my fault) in the test file are holding up the Travis CI build process.

TypeScript n00b here:
I had a bunch of sections like the following in my project:
declare module "meteor/meteor" {
   let Meteor: any
   export Meteor
}

This let me import {Meteor} from "meteor/meteor" until the definitions
were updated. Now that they are, when I install this module and remove
the above block, I still get:
server/stripe.ts (1, 22): Cannot find module 'meteor/meteor'.
What am I not getting this time? :) Do these type definitions not
include what is needed to import Meteor modules using ES2015 syntax?
Thanks.

@ndarilek it's not updated yet

Ah, got it. I did "typings install --ambient --save meteor" and got a
definition that claims to be for 1.3. Also, the deprecation warning went
away. Am I installing the wrong definition, or is the update only partial?
Thanks.

@ndarilek  Here is that PR https://github.com/DefinitelyTyped/DefinitelyTyped/pull/9113.
It needs an approval from couple of more people.
I think that claim is wrong, it's still for Meteor 1.2.

Yes -- the 1.3 claim is wrong.  The previous definitions were automatically generated from the official Meteor 1.3 JSDocs JSON file, but the necessary additions were not made for Meteor 1.3 imports and exports.

@fullflavedave I've recently started moving Meteor and major packages' typings to dedicated repos under one organization, i.e. check out https://github.com/meteor-typings and https://github.com/meteor-typings/meteor for more info. I think it's time to deprecate https://github.com/meteor-typescript/meteor-typescript-libs since most Meteor specific typings have been moved to my mind, while others - either deprecated or third part libs typings, which already exist and is maintained in other registries (either DefinitelyTyped or typings's registry).

Great work!  I'll deprecate https://github.com/meteor-typescript/meteor-typescript-libs and reference the typings and DefinitelyTyped registries in the README.
I'd love to help out with typings, since I use TypeScript and Meteor everyday for my work and occasionally find random improvements for the definitions.
Thanks for all of your good work on this!

Looks good, will give it a try in a few weeks when I can find free time.
Olivier
On Wednesday, June 8, 2016 5:59 AM, Alex<EMAIL_ADDRESS>wrote:

@fullflavedave I've recently started moving Meteor and major packages' typings to dedicated repos under one organization, i.e. check out https://github.com/meteor-typings and https://github.com/meteor-typings/meteor for more info. I think it's time to deprecate https://github.com/meteor-typescript/meteor-typescript-libs since most Meteor specific typings have been moved to my mind, while others - either deprecated or third part libs typings, which already exist and is maintained in other registries (either DefinitelyTyped or typings's registry).—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

I just saw this announcement:  https://blogs.msdn.microsoft.com/typescript/2016/06/15/the-future-of-declaration-files/.  So would it make sense to try to point the meteor definitions at https://www.npmjs.com/package/@types/meteor to the newest meteor-typings definitions or temporarily update the definitions on the DefinitelyTyped branch that it's pulling from (or perhaps both)?

@fullflavedave It seems to be intended for NPMs. Since Meteor doesn't come as NPM, not sure it's going to work for it.  typings for example is much more flexible in that providing different installation options, including separation of the browser and node versions.

Discussions/issues from now on are here - https://github.com/barbatus/typescript


fixed sudo, mode and newline in LineIntoFile

new pull req that fixes all issues with LineIntoFile. (see also #56)
I realise that I probably should just have fixed the #56 one and not start over again. sorry for that.
Alex

I'd rather see the it accept exec options than always using sudo:
func (c Linux) LineIntoFile(h Host, path, matcher, newLine string, opts ...exec.Option) error {
   return h.Exec(whatever, opts...)
}

Even better would be to reimplement it completely in Go (read file, modify in memory, write back) as the TODO says:
// TODO refactor this into go because it's too magical.

I don't really see the problem. If this would be done entirely in go (which would indeed be better), it would effectively run as 'sudo' as well, since the new file that is created if one did not exist before, is also owned by root.
I'm not a go programmer so I can't make your clean solution, I'm just trying to help and fix something that is currently broken.

I don't really see the problem. If this would be done entirely in go (which would indeed be better), it would effectively run as 'sudo' as well, since the new file that is created if one did not exist before, is also owned by root.

That's where the exec options come to play, this also applies to the shell version.

Doing that will require altering WriteFile or writing the new file in the shell snippet.

I just noticed there's no windows LineIntoFile at all, that's a bit bad for compatibility :( I'll look into it.


Fix the padding values in the edit mode input.

The previous values caused layout thrashing when transitioning between edit and label mode.

Thanks :)


Git: Initialize Repository button does not refresh state if git.autoRepositoryDetection is set to false

Does this issue occur when all extensions are disabled?:

Yes

VS Code Version:
Version: 1.77.1
Commit: b7886d7461186a5eac768481578c1d7ca80e2d21
Date: 2023-04-04T23:21:11.906Z
Electron: 19.1.11
Chromium: 102.0.5005.196
Node.js: 16.14.2
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.22621
Sandboxed: No

OS Version:

Microsoft Windows 11 Pro
10.0.22621 N/A Build 22621

Description:
Not sure if this is intended behavior or not but I think if I use the Initialize Repository button it should create and show the repository.
If git.autoRepositoryDetection is set to false, the repository is actually created but nothing is shown in VsCode
I understand that this option is for auto-detect and it's kind of logical, but should it affect the Initialize Repository action?
Steps to Reproduce:

@kanlukasz, thank you very much for filing this issue. Setting git.autoRepositoryDetection to false disables repository detection independent whether it already exists or it has been created using VS Code. While this is the intended behaviour, I agree that we could probably make this experience better.


[BugFix] ensure a valid cast clause for MySQL/JDBC table (backport #23379)

Fixes #issue
What type of PR is this:

[x] BugFix
[ ] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[x] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[ ] 3.1
[ ] 3.0
[ ] 2.5
[ ] 2.4

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[FE Incremental Coverage Report]
:heart_eyes: pass : 25 / 31 (80.65%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/analysis/SlotRef.java
3
8
37.50%
[272, 292, 299, 300, 302]

:large_blue_circle:
com/starrocks/sql/optimizer/rewrite/ExternalTablePredicateExtractor.java
16
17
94.12%
[78]

:large_blue_circle:
com/starrocks/sql/optimizer/rule/transformation/PushDownPredicateToExternalTableScanRule.java
2
2
100.00%
[]

:large_blue_circle:
com/starrocks/load/Load.java
1
1
100.00%
[]

:large_blue_circle:
com/starrocks/sql/plan/ScalarOperatorToExpr.java
1
1
100.00%
[]

:large_blue_circle:
com/starrocks/analysis/CastExpr.java
2
2
100.00%
[]

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)


Doom installation is successful, but there is no beautiful interface of doom

➜  .emacs.d git:(develop) ✗ bin/doom sync

Synchronizing your config with Doom Emacs...
Regenerating envvars file at "~/.emacs.d/.local/env"
- Ignoring PWD=/home/kehenghao/.emacs.d
- Ignoring GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1
- Ignoring SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
- Ignoring TERM=xterm-256color
- Ignoring DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
- Ignoring HOME=/home/kehenghao
- Ignoring SSH_AGENT_PID=2068
✓ Successfully generated "~/.emacs.d/.local/env"
Installing packages...
- No packages need to be installed
(Re)building packages...
✓ No packages need rebuilding
Purging orphaned packages (for the emperor)...
- No builds to purge
- Skipping elpa packages
- Skipping repos
- Skipping regrafting
(Re)generating autoloads file...
Generating autoloads file...
Byte-compiling autoloads file...
✓ Generated .local/autoloads.elc

Restart Emacs or use 'M-x doom/reload' for changes to take effect
✓ Finished! (2.4985s)

emacs alse is The most primitive interface, doom don't work.
➜  .emacs.d git:(develop) ✗ uname -a
Linux kecode 5.0.0-32-generic #34~18.04.2-Ubuntu SMP Thu Oct 10 10:36:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
What should I do? Can you give me some guidance？3q.

https://github.com/hlissner/doom-emacs/blob/develop/docs/faq.org#i-get-the-vanilla-emacs-splash-screen-at-startup


Customer Interviews

Work together with platform engineering on customer interviews and incorporating their input on roadmap requests and documentation as indicated.

closing due to age


Link failure (CCLD pq) building versions 3.2.1/3.2.3 on 64-bit raspberry pi

OS is Debian GNU/Linux 11 (bullseye) on 64bit raspberry pi 4b aarch64 GNU/Linux and I'm building GAP 4.11.1 thereupon. Everything gappy makes ok except for half a dozen outliers amongst the pkg inclusions, all of which are fixable except, sadly, anupq which fails to complete the build after its successful compilations.
This package makes fine on both a 32 bit raspberry pi under standard (i.e. not raspbian) ubuntu and also a 64 bit x86amd64 system (i.e. not even a raspberry pi). As a matter of possible interest I've tried making the most recent
version (anupq 3.2.3) and exactly the same link fails turn up.
Something about raspberry pi 64bit is wonky, wrt certain declared variables, and I'm getting
  CCLD     pq
/usr/bin/ld: src/pq-CloseFile.o:/home/paul/gittage/gap-4.11.1/pkg/anupq-3.2.1/./include/pcp_vars.h:81:
	multiple definition of `y_address'; src/pq-AllocateSpace.o:/home/paul/gittage/gap-4.11.1/pkg/anupq-3.2.1/./include/pcp_vars.h:81: first defined here
/usr/bin/ld: src/pq-Extend_Auts.o:/home/paul/gittage/gap-4.11.1/pkg/anupq-3.2.1/./include/pcp_vars.h:81:
	multiple definition of `y_address'; src/pq-AllocateSpace.o:/home/paul/gittage/gap-4.11.1/pkg/anupq-3.2.1/./include/pcp_vars.h:81: first defined here

(a whole slew of over 100 similar messages about variables Group_library, Compact_Description, Compact_Order, Group_library_file, y_address ending with ...)
/usr/bin/ld: src/pq-main.o:/home/paul/gittage/gap-4.11.1/pkg/anupq-3.2.1/./include/global.h:17:
	multiple definition of `Group_library'; src/pq-GAP_link_via_file.o:/home/paul/gittage/gap-4.11.1/pkg/anupq-3.2.1/./include/global.h:17: first defined here
/usr/bin/ld: src/pq-main.o:/home/paul/gittage/gap-4.11.1/pkg/anupq-3.2.1/./include/pcp_vars.h:81:
	multiple definition of `y_address'; src/pq-AllocateSpace.o:/home/paul/gittage/gap-4.11.1/pkg/anupq-3.2.1/./include/pcp_vars.h:81: first defined here
collect2: error: ld returned 1 exit status
make: *** [Makefile:1001: pq] Error 1

Doubtless something silly I've forgotten. Any ideas why this bunch would fail to link on a 64 bit raspi where they do on a 32 bit raspi or a 64 bit non-raspi?

How do you build it? Just plain ./configure && make ?

Yup. Plain old. Works fine on the other two machines. Need a config.log?

Yes, please

config.log
note possible distraction from grepping the above for 'fatal' which yields
gcc: fatal error: no input files
gcc: fatal error: no input files
conftest.c:11:10: fatal error: ac_nonexistent.h: No such file or directory
conftest.c:11:10: fatal error: ac_nonexistent.h: No such file or directory
conftest.c:62:10: fatal error: vfork.h: No such file or directory
conftest.c:29:10: fatal error: vfork.h: No such file or directory

but the exact same obtains for the config.log files from the other two successful builds.
And thanks for your attention btw!

The messages about vfork.h etc. are normal and harmless.
By chance we have two Raspberry Pis set up in our research group, one with 32bit and one with 64bit.
I can reproduce the issue on the 32bit one, which runs Raspbian GNU/Linux 11 (bullseye) with gcc (Raspbian 10.2.1-6+rpi1) 10.2.1 20210110.
It works fine on the 64bit one, which runs Ubuntu 20.04.3 LTS (Focal Fossa) with gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0.
In your config log, it says that it uses gcc (Debian 10.2.1-6) 10.2.1 20210110.
Perhaps a compiler bug?

Could also be an issue with ld, of course.
On the working 64 bit system, ld --version reports GNU ld (GNU Binutils for Ubuntu) 2.34
On the broken 32 bit system it reports GNU ld (GNU Binutils for Raspbian) 2.35.2.

#53 happens to fix the multiple definition problem (I also found it in cygwin). I believe the code is broken by the C standard (it violates the one definition rule), but many linkers "helpfully" handle this case by merging the variables, particularly if they take the value 0.


How to know the starting position of string in JSON using the prason library?

If I want to get the position of string 'data' in the JSON array rxpk, how to program it?
I know that this library can get the content with code below, but I can't find the function of the position.
const char* data= json_object_get_string(json_array_get_object(json_object_get_array(json_value_get_object(json_parse_string_with_comments((const char*)(original string))), "rxpk"), 0), "data");
All I want to do is replace the original content with a new string to fill in.

你应该用json_object_set_string.

你应该用json_object_set_string.

In this case, the content of the JSON object is changed, but what if I want to use the whole changed JSON?
Is there a way to turn the changed JSON object into a char array?
Thanks a lot.

You need to parse it, modify it and serialise it. This kind of questions belong on stack overflow.

You need to parse it, modify it and serialise it. This kind of questions belong on stack overflow.

I'm sorry, but now I have an idea of the whole process.
When using json_serialize_to_string or json_serialize_to_string_pretty, there are often some mistakes, such as converting / to \ / or adding decimal places after some big number. I will try to solve these problems.
Thanks again.


bitrise share start failing

I am running the bitrise share start -c command with a fork of this repository as input and I keep getting the following error
"FATA[19:52:45] Failed to save step spec, error: Failed to walk through path, error: not supported key found in options: is_sensitive "
A lot of steps in this repository use the is_sensitive option, but I am not sure what I should do about it. I just want to publish my step 😅
I am using bitrise cli version 1.42.0
My fork can be found at https://github.com/realappie/bitrise-steplib

This is a duplicate of https://github.com/bitrise-io/bitrise/issues/733. Responding on that issue.


farm with poll limit 0 dont stops

For unit test i would like to farm with poll limit 0. I expect 1 ping and than it should stop. For some reason it does not. I will fix it and add some unit test later but at the moment i cant merge to much together.

Bugfix ready. I will check the unit test again as soon as pull request https://github.com/Storj/dataserv-client/pull/131 is closed.

Retest ok.


BUG: Inconsistent margins and paddings

Is there an existing issue for this?

[X] I have searched the existing issues

What went down?
Inconsistent Margins and Padding Across Components
After reviewing the site, I noticed inconsistencies in the margins and padding across different components, which affect the overall layout and user experience.
Example 1:

Example 2:

I've attached screenshots for better visualization of the issue.
Proposal: I would like to work on this issue to standardize the margins and padding for all components, ensuring consistency throughout the site.
Kindly assign the appropriate labels, as I am a contributor for GSSOC 2024.
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a gssoc-24-extd contributor
[X] I want to work on this issue
[X] I check the issue doesn't exist

Its not a big issue you can work upon another issues

@RamakrushnaBiswal  what is a problem with you? you even closed my issue now? are you really expecting every issue should be of high level difficulty?

@tejasbenibagde for me the margin and padding is looks good

@RamakrushnaBiswal  how can you say this without even looking at my updated code ? did you ever compared it?

@RamakrushnaBiswal  sorry but I'm reporting you to the GSSOC community.

Ok go for no issues @tejasbenibagde

@RamakrushnaBiswal how can you say this without even looking at my updated code ? did you ever compared it?

Well I looked into that you change too many files

@RamakrushnaBiswal  You are correct I changed too many files cause I worked for hours doing that just for you to ignore everything in seconds.

@RamakrushnaBiswal You are correct I changed too many files cause I worked for hours doing that just for you to ignore everything in seconds.

I am really sorry about that you should follow the rules

Create a issue
Wait for project admin assign
Then go for work

I think it's your 1st time in open source

It is not my first time doing it but I understand your point of view now. Sorry I went hyper all of a sudden.

@RamakrushnaBiswal  While working on the project, I noticed several optimization and modularization issues that are impacting the site's performance. I believe I can address these issues to improve the overall speed and efficiency of the application.
Would you like me to create a new issue for this? If so, could you please assign it to me? I want to mention that this may involve changes to multiple files, but I’ll ensure everything is well-documented.


[WIP] Use UI-Select for Choice and Choices fields

It adds select filtering and a better UI for multi-selects.
Before:

Now:

It will especially be useful to populate live select lists using AJAX requests.
Tasks:

[x] Implement UI-Select library
[ ] Update or create tests to ensure we use this lib

Promising!

Miss a last point on this PR: Prevent ENTER from submitting the form when in select list. Working on it. As I'm not sure if it depends from ng-admin or from ui-select, switching this one as RFR.

Back to WIP: multiple choices required constraint is not taken into account.

Back to RFR.


Changes when trying to run finetuning on cluster

@lintangsutawika

@haileyschoelkopf would it make more sense to change train_mtf to mean packing, from the proposed changes here, it seems that the only time train_mtf is used is in training.py and for deciding if position_ids should be taken from the dataset or should be calculated there.


arterial thrombosis

Add new term:
abnormal arterial thrombosis (a child of abnormal thrombosis MP:0005048)
DEF: any anomaly in the formation or presence of thrombi inside one or more arteries, causing the obstruction of blood flow
SYN: abnormal artery thrombosis
SYN: blood clot in one or more arteries

Added new term:
abnormal arterial thrombosis MP:0031157


Record Button Functionality

I’ve thought about the functionality of the Record Button for awhile now. I think it would work better if the Record Button wasn’t just an “on/off button” but a “latch button”. When you press it, it would depress (latch inward) and stay depressed. At this point it would not start recording until you hit the play button in the host (Reaper). When the host stops playing the track the record button would unlatch outward (stop recording inside the plugin and turn off).

I wish the record button would also post a parameter in the plugins parameter list so I could activate it from the host (Reaper).

If you are done reviewing and setting the parameters of the plugin with the imported track into the plugin. Then when the Record button is in the unlatched position it acts like a normal plugin and always shoots the midi output of the plugin to the current track in the DAW so it can route it to a empty midi track below the current track when the DAW’s play button is hit. This process would complete the process of recording sound into the plugin, refining it then be set in the right position to record its output back into the DAW with a Record button that latches and has two functions.


hopper minecart cannot insert items into engine

make hopper minecarts work with the ship engine

While testing with Eureka 1.5.0 on Minecraft 1.20.1 I find the following

you can extract using a hopper or hopper minecart
you cannot transfer between ship and non-ship using a hopper minecart (not a bug)
hopper minecart cannot insert items (this is a bug)

While testing with Eureka 1.5.0 on Minecraft 1.20.1 I find the following

you can extract using a hopper or hopper minecart
you cannot transfer between ship and non-ship using a hopper minecart (not a bug)
hopper minecart cannot insert items (this is a bug)

Number 3 does not occur on 1.18.2.

While testing with Eureka 1.5.0 on Minecraft 1.20.1 I find the following

you can extract using a hopper or hopper minecart
you cannot transfer between ship and non-ship using a hopper minecart (not a bug)
hopper minecart cannot insert items (this is a bug)

Number 3 does not occur on 1.18.2.

Just realized that hopper minecarts cannot insert items. They aren't supposed to do that. I was thinking of chest minecarts. Number 3 is not a eureka issue.

Oh yeah, and that means this issue isn't a bug either. Someone close it

oh you are right


.env file is public

you must add a .ignore file  with a .env in it so you .env file will be hidden in the repo

Thanks very much.
On Thu, 22 Dec 2022 at 23:50, Sohaib manah @.***> wrote:

you must add a .ignore file with a .env in it so you .env file will be
hidden in the repo
—
Reply to this email directly, view it on GitHub
https://github.com/jaymeeu/mdsearch_backend/issues/1, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AMJDRQFZDZRBKHY72QVXIZ3WOTSMFANCNFSM6AAAAAATHF6KYA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


Update to 1.87

https://github.com/ocornut/imgui/releases/tag/v1.87
There are some breaking changes and the generator is failing:
generator/DearImGui/Generator.hs:72:11: error:
    • Exception when trying to run compile-time code:
        Couldn't parse Dear ImGui headers:

offset=9459:
Couldn't lookup the following values in enum ImGuiKey_:
 - ImGuiKey_COUNT

Number 512.0 ""
ReservedSymbol ','
Identifier "ImGuiKey_NamedKey_END"
ReservedSymbol '='
Identifier "ImGuiKey_COUNT"

ReservedSymbol ','
Identifier "ImGuiKey_NamedKey_COUNT"
ReservedSymbol '='
Identifier "ImGuiKey_NamedKey_END"
Symbolic "-"
Identifier "ImGuiKey_NamedKey_BEGIN"
ReservedSymbol ','
BeginCPP "ifdef"
Identifier "IMGUI_DISABLE_OBSOLETE_KEYIO"
EndCPPLine
Identifier "ImGuiKey_KeysData_SIZE"
ReservedSymbol '='
Identifier "ImGuiKey_NamedKey_COUNT"
ReservedSymbol ','
Comment "Size of KeysData[]: only hold named keys"

CallStack (from HasCallStack):
  error, called at generator/DearImGui/Generator.hs:87:9 in dear-imgui-1.3.1-inplace-dear-imgui-generator:DearImGui.Generator
      Code: do currentPath <- TH.loc_filename <$> TH.location
               basicHeaders <- TH.runIO
                                 do headersPath <- canonicalizePath
                                                     (takeDirectory currentPath
                                                        <> "/../../imgui/imgui.h")
                                    ....
               TH.lift $ generateNames basicHeaders
    • In the untyped splice:
        $(do currentPath <- TH.loc_filename <$> TH.location
             basicHeaders <- TH.runIO
                               do headersPath <- canonicalizePath
                                                   (takeDirectory currentPath
                                                      <> "/../../imgui/imgui.h")
                                  headersSource <- Text.readFile headersPath
                                  ....
             TH.lift $ generateNames basicHeaders)
   |
72 | headers = $( do
   |           ^^^^^...

Closedwith #125


[FEATURE] Manage Spark History Server as a deployment via the Spark Operator helm chart.

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?
The Spark History server is a valuable debugging and process tracing tool.  Currently, deployment of the history server would have to occur independently from the operator.  It would be a convenience to manage the Spark History Server (SHS) via the Spark Operator helm chart.
Describe the solution you would like
A new section shall be added to the spark operator helm chart to define parameters for the SHS deployment.  We note that a confounding element of this feature is storage layers.  SHS is dependent on some accessible storage layer where spark logs can be found.  The simplest implementation is a shared NFS volume, but blob storage such as S3 or an Azure storage account are common solutions that should be easy to use with our implementation.  These third party solutions require additional libraries to be loaded into the classpath-- a task that SHS fails to trivialize.
Describe alternatives you have considered
The alternative involves individuals rolling their own deployments for SHS-- a non-trivial process.
Additional context
If we choose to pursue this, we may also wish to consider managing deployment of the Hive Thrift Server.

Suggested Architecture

SHS will optionally be deployed in a sidecar container running alongside the Operator.
In order to resolve the problem of dynamically pulling in dependencies/packages, an initcontainer shall be spun up which populates a volume with the union of the default $SPARK_HOME/jars and the result of java -Divy.cache.dir=$SPARK_HOME -Divy.home=$SPARK_HOME -jar $SPARK_HOME/jars/ivy-2.5.1.jar -dependency [PACKAGE].  This populated volume shall be mounted in the SHS container as $SPARK_HOME/jars
$SPARK_HOME/conf/spark.conf shall be mounted as a volume populated by a raw text block in the helm chart.

Did some initial work on this just to feel it out-- Got automatic resolution of packages working via initcontainers.  It's a bit gross, but it works as a start.
Major TODO items:
[ ] Add arbitrary volume/volumeMount support
[ ] Add support for pulling jars, rather than solely packages
[ ] Add a clean mechanism for mounting spark-defaults.conf
[ ] Create an example that works out of the box-- The hard part being a zero-barrier-to-entry Volume accessible across nodes.
[ ] Docs updates
[ ] General cleanup / hardening

I'm not sure if it's a good idea to have history server co-deployed with operator. A single history server can aggregate jobs managed by multiple Spark operator deployments across multiple k8s clusters

I think the real problem here isn't so much that operator should be managing the history server directly, and more that history server, a valuable part of the spark ecosystem, doesn't have any good helm charts out in the wild.

I agree. I haven't looked at the quality of https://artifacthub.io/packages/helm/cloudnativeapp/spark-history-server, but if it's something you have used, I'm for that idea

I'm not sure if it's a good idea to have history server co-deployed with operator. A single history server can aggregate jobs managed by multiple Spark operator deployments across multiple k8s clusters

I think the real problem here isn't so much that operator should be managing the history server directly, and more that history server, a valuable part of the spark ecosystem, doesn't have any good helm charts out in the wild.

I agree. I haven't looked at the quality of https://artifacthub.io/packages/helm/cloudnativeapp/spark-history-server, but if it's something you have used, I'm for that idea

Sounds reasonable to me. Wrt the helm chart I linked, I wasn't sure if you had specific thoughts, given that you're listed as the maintainer.

Ah upon a closer look, now I remember that I initially created this chart many years ago. I haven't used it for a long time though and won't count on it still being production ready

I think there's both interest and clearly an unfilled need in the community for a production ready, standalone spark history chart that's well maintained. Would kubeflow and the spark operator maintainers be open to one being created in this repo, or would it be better housed somewhere totally separate?

Kindly make the spark history server part of the operator.
I think targeting this operator as single point for spark on K8s eco system will add much better momentum for the development.
For example integrating spark operator to manage an external shuffle service on K8s.
Sorry for interrupting but I am so excited about the new development on this operator

I am also looking forward to a well maintained helm chart for spark history server, and I think maybe spark operator
repo is the best place to host this chart. Would you @yuchaoran2011  mind me contributing a new helm chart based on this one  https://artifacthub.io/packages/helm/cloudnativeapp/spark-history-server and put it under charts/spark-history-server.

I noticed that @vara-bonthu had maintained one helm chart for spark history server https://github.com/KubedAI/spark-history-server with support for S3. And I want to know what do you think about creating a new one for history server in this repo?

Spark History Server isn't directly tied to the Spark Operator project. It's usually deployed by users on Kubernetes, even if they don't use the Spark Operator. For example, users running spark-submit without the operator often set up the Spark History Server on their own. This is a separate deployment and, for large workloads, might need multiple replicas. So, it doesn't make sense to link it directly to the Spark Operator project.
If the community is interested, we could propose making the Spark History Server its own project. This could be under Kubeflow or Apache, focusing on multi-cloud and self-managed setups.
This PR can be moved to a new repo.


State clean up

Codecov Report

Merging #442 (a1e0ff1) into main (9e59170) will decrease coverage by 0.1%.
The diff coverage is 0.0%.

@@           Coverage Diff           @@
##            main    #442     +/-   ##
=======================================
- Coverage   13.1%   13.0%   -0.1%
=======================================
  Files         27      27
  Lines       7875    7918     +43
=======================================
  Hits        1031    1031
- Misses      6844    6887     +43

Impacted Files
Coverage Δ

src/swapd/runtime.rs
0.0% <0.0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9e59170...a1e0ff1. Read the comment docs.

None of the swap functional tests pass, neither in the CI nor on my local machine. On main they run fine. The monero syncer address lws test is expected to fail until #448 is fixed by #444 .

it works until 9ec26af so the issue is local_params that is not present long enough somewhere and contains the destination address


Add features on Home Assistant Script

Bugfixes:

Allow home assistant instances on different port than 8123
allow home assistant instances with SSL access (SSL connections verified or not)
capture exceptions on getting data from i2c bus
entities (states) are created under the "sensor" integration, to allow Home Assistant to gauge them

Functionalities:

entities grouped under the same unique_id
use logging module instead of print to allow logging level selection and logging on journal when daemonized

Thanks, we will review and consider for the next release.