Blueprint::shardKey does not exist.

Getting this error while trying to migrate!
`2014_10_12_000000_create_users_table .................................................................................................... 3ms FAIL
BadMethodCallException
Method Illuminate\Database\Schema\Blueprint::shardKey does not exist.`

php -v
PHP 8.1.8 (cli) (built: Aug  1 2022 18:56:21) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.8, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.8, Copyright (c), by Zend Technologies

php artisan --version
Laravel Framework 9.26.0

Can you show me your database.php file? Laravel should be giving you an instance of SingleStore\Laravel\Schema\Blueprint not Method Illuminate\Database\Schema\Blueprint.

Here it is
`<?php
use Illuminate\Support\Str;
return [
/*
|--------------------------------------------------------------------------
| Default Database Connection Name
|--------------------------------------------------------------------------
|
| Here you may specify which of the database connections below you wish
| to use as your default connection for all database work. Of course
| you may use many connections at once using the Database library.
|
*/

'default' => env('DB_CONNECTION', 'singlestore'),

/*
|--------------------------------------------------------------------------
| Database Connections
|--------------------------------------------------------------------------
|
| Here are each of the database connections setup for your application.
| Of course, examples of configuring each database platform that is
| supported by Laravel is shown below to make development simple.
|
|
| All database work in Laravel is done through the PHP PDO facilities
| so make sure you have the driver for your particular database of
| choice installed on your machine before you begin development.
|
*/

'connections' => [

    'singlestore' => [
        'driver' => 'singlestore',
        'url' => env('DATABASE_URL'),
        'host' => env('DB_HOST'),
        'port' => env('DB_PORT'),
        'database' => env('DB_DATABASE'),
        'username' => env('DB_USERNAME'),
        'password' => env('DB_PASSWORD'),
        'unix_socket' => env('DB_SOCKET'),
        'charset' => 'utf8mb4',
        'collation' => 'utf8mb4_unicode_ci',
        'prefix' => '',
        'prefix_indexes' => true,
        'strict' => true,
        'engine' => null,
        'options' => extension_loaded('pdo_mysql') ? array_filter([
            PDO::MYSQL_ATTR_SSL_CA => env('MYSQL_ATTR_SSL_CA'),
            PDO::ATTR_EMULATE_PREPARES => true,
            PDO::ATTR_PERSISTENT => true
        ]) : [],
    ],

    'sqlite' => [
        'driver' => 'sqlite',
        'url' => env('DATABASE_URL'),
        'database' => env('DB_DATABASE', database_path('database.sqlite')),
        'prefix' => '',
        'foreign_key_constraints' => env('DB_FOREIGN_KEYS', true),
    ],

    'mysql' => [
        'driver' => 'mysql',
        'url' => env('DATABASE_URL'),
        'host' => env('DB_HOST', '<IP_ADDRESS>'),
        'port' => env('DB_PORT', '3306'),
        'database' => env('DB_DATABASE', 'forge'),
        'username' => env('DB_USERNAME', 'forge'),
        'password' => env('DB_PASSWORD', ''),
        'unix_socket' => env('DB_SOCKET', ''),
        'charset' => 'utf8mb4',
        'collation' => 'utf8mb4_unicode_ci',
        'prefix' => '',
        'prefix_indexes' => true,
        'strict' => true,
        'engine' => null,
        'options' => extension_loaded('pdo_mysql') ? array_filter([
            PDO::MYSQL_ATTR_SSL_CA => env('MYSQL_ATTR_SSL_CA'),
        ]) : [],
    ],

    'pgsql' => [
        'driver' => 'pgsql',
        'url' => env('DATABASE_URL'),
        'host' => env('DB_HOST', '<IP_ADDRESS>'),
        'port' => env('DB_PORT', '5432'),
        'database' => env('DB_DATABASE', 'forge'),
        'username' => env('DB_USERNAME', 'forge'),
        'password' => env('DB_PASSWORD', ''),
        'charset' => 'utf8',
        'prefix' => '',
        'prefix_indexes' => true,
        'search_path' => 'public',
        'sslmode' => 'prefer',
    ],

    'sqlsrv' => [
        'driver' => 'sqlsrv',
        'url' => env('DATABASE_URL'),
        'host' => env('DB_HOST', 'localhost'),
        'port' => env('DB_PORT', '1433'),
        'database' => env('DB_DATABASE', 'forge'),
        'username' => env('DB_USERNAME', 'forge'),
        'password' => env('DB_PASSWORD', ''),
        'charset' => 'utf8',
        'prefix' => '',
        'prefix_indexes' => true,
        // 'encrypt' => env('DB_ENCRYPT', 'yes'),
        // 'trust_server_certificate' => env('DB_TRUST_SERVER_CERTIFICATE', 'false'),
    ],

],

/*
|--------------------------------------------------------------------------
| Migration Repository Table
|--------------------------------------------------------------------------
|
| This table keeps track of all the migrations that have already run for
| your application. Using this information, we can determine which of
| the migrations on disk haven't actually been run in the database.
|
*/

'migrations' => 'migrations',

/*
|--------------------------------------------------------------------------
| Redis Databases
|--------------------------------------------------------------------------
|
| Redis is an open source, fast, and advanced key-value store that also
| provides a richer body of commands than a typical key-value system
| such as APC or Memcached. Laravel makes it easy to dig right in.
|
*/

'redis' => [

    'client' => env('REDIS_CLIENT', 'phpredis'),

    'options' => [
        'cluster' => env('REDIS_CLUSTER', 'redis'),
        'prefix' => env('REDIS_PREFIX', Str::slug(env('APP_NAME', 'laravel'), '_').'_database_'),
    ],

    'default' => [
        'url' => env('REDIS_URL'),
        'host' => env('REDIS_HOST', '<IP_ADDRESS>'),
        'username' => env('REDIS_USERNAME'),
        'password' => env('REDIS_PASSWORD'),
        'port' => env('REDIS_PORT', '6379'),
        'database' => env('REDIS_DB', '0'),
    ],

    'cache' => [
        'url' => env('REDIS_URL'),
        'host' => env('REDIS_HOST', '<IP_ADDRESS>'),
        'username' => env('REDIS_USERNAME'),
        'password' => env('REDIS_PASSWORD'),
        'port' => env('REDIS_PORT', '6379'),
        'database' => env('REDIS_CACHE_DB', '1'),
    ],

],

];
`

Here is the database.php file of mine!
https://gist.github.com/rana01645/838000e5265ec1c28338064588b22155
By default database is importing Illuminate\Database\Schema\Blueprint, I also tried by using SingleStore\Laravel\Schema\Blueprint one, another issue arised like Schema::create is expecting the laravel's default one!

@rana01645 excellent, that looks right to me!
Is there a DB_CONNECTION var in your .env?

DB_CONNECTION was previously set to mysql in the .env file
After changing it to
DB_CONNECTION=singlestore
Seems like it's working fine! Thanks.

Btw, I'm having hard time setting up github workflow with singlestore for testing the db in Laravel. The workflow example provided for nodejs is working fine! (https://github.com/rana01645/singlestore-and-github-actions-demo)
Where should I open a issue for this?

Glad we got that fixed!
For the github workflow, are you mirroring the one in this repo (https://github.com/singlestore-labs/singlestoredb-laravel-driver/blob/main/.github/workflows/tests.yml) and having issues?

No, I was trying with the node js one (https://github.com/rana01645/singlestore-and-github-actions-demo/blob/main/.github/workflows/singlestore-github-actions.yml)
Thanks for providing this one, It fixed my issue with the workflow!

Excellent, glad to hear it. Let me know if you run into anything else.

Btw, I created a visible integration for Singlestore with a fresh Laravel project!
It also covers the automated database testing for Singlestore database with GitHub workflow.
Hope it will be helpful for someone who wants to quickstart Singlestore with Laravel.
https://github.com/rana01645/laravel-singlestore
Pull request with the necessary changes to get started
https://github.com/rana01645/laravel-singlestore/pull/1

@aarondfrancis
Is there any guideline related to using Singlestore in laravel sail?

Getting SingleStore running locally is outside the scope of this package, unfortunately. I have not used Laravel Sail!


Could not get it to work

[Enter feedback here]
Something wrong with ports donno and don't care, too much hustle, ended up creating a .net core 5 asp react app with redux, where only one comma has to be deleted from package.json for the project to run [sic.]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f5beb7f5-5d28-d5e5-92ba-3e04c75e44b5
Version Independent ID: 867e3a3f-2387-ba46-c911-10822f8c4a81
Content: Create an ASP.NET Core app with React - Visual Studio (Windows)
Content Source: docs/javascript/tutorial-asp-net-core-with-react.md
Product: visual-studio-windows
Technology: vs-javascript
GitHub Login: @Mikejo5000
Microsoft Alias: mikejo

@fashrista -- Jib, thank you for your feedback.  Can you provide more details?  Which comma needs to be deleted?  Meanwhile, please consider these resources:

Visual Studio on Microsoft Q&A
Visual Studio MSDN Forum
Stack Overflow
Reddit

@fashrista Navigating with your ports is explained in this section here
If it is still not working.. Please use the Report a Problem tool to report the issue. When you do this, your report becomes part of the work items that are listed in the in the Visual Studio Developer Community, which allows you to engage directly with the engineering team while they address and fix the problem.

@gcrev93 -- Gabrielle, thank you for clarifying.
I am closing this issue now.  @fashrista -- Jib, you are welcome to @ mention me for any followup.  We hope to hear from you again.


Specify format for descriptive material

Often datasets provide descriptions, e.g. WALS' chapter texts, which reference parts of the dataset, e.g. the set of values of a WALS feature. It would be good to have a specification how such material can/should be added to a CLDF dataset. Considering that with CLDF we can already associate a valueUrl with each row of each table, it seems natural to chose HTML as supported format for descriptions (or markdown) - using standard HTML links to mark up references.

We've said that the github wiki from glottobank/grambank will be made public, along with github issue threads tagged for feature discussion. The question then is how to best relate that information to the cldf ecology.

sorry for interfering
Since I don't know any solution for automatic import and export wiki pages from github, I'd prefer markdown. Otherwise our solution will be stick with one particular platform.

@agricolamz Problem is with documents that already are HTML. "HTML to markdown" isn't fully specified, whereas "markdown to HTML" is. But anyway, allowing both shouldn't be much of a problem, as long as it is clear that the way to reference data is via URLs in links - whether these are specified as HTML "a" tags or in markdown shorthand [](...).

So while I envisage applications like static-site generators for CLDF datasets to support descriptive material of this sort sometime down the road, the main point now is to specify how bits and pieces in CLDF datasets can be referenced from documents - and I don't see a better way than to exploit the valueUrl option.

How would we reference sources, i.e. entries in the bib file? Assign an artificial valueUrl of /sources/<bibtex-key>? Or use using the name of the bibtex file as path of the URL, i.e. sources.bib/<bibtex-key>? If the latter, we might as well fall back to <table-url>/<id> as default valueUrl for rows in data files.

CLDF markdown may be a solution here.

Using wiki articles and pulling those into the parameters table has worked reasonably well for Grambank, with some kerfuffle sometimes with some formatting. Is that something future databases are likely to do too?

Including multiline text content in CSV clearly adds some complexity to downstream processing, so having an alternative method to include descriptive material may be useful. With MediaTable, we already have a way to associate any CLDF object with a file. So if this file was specified with mimetype text/markdown and an additional CLDF property derived from dc:format to say it's "CLDF markdown", downstream processors could render the material on demand.

Superseded by CLDF Markdown, see #131


Ignore AntiForgeryToken

This is a suggestion based on multiple issues surrounding situations where you have load balanced environments or other scenarios causing the following error: System.Security.Cryptography.CryptographicException: The key {0b722c3b-499f-4fc5-a4cc-f70f94b007f8} was not found in the key ring.
Setting IgnoreAntiforgeryToken = true removes the error (obviously).
The M$ suggested resolution is to sync the data protection folders across instances which might not always be feasible (plus it still doesn't work for me).
Since this plugin already puts the dashboard behind Umbraco authentication I figure this can be left out.

Thanks @jrunestone! I don't know if you participated in Digital Ocean's Hacktoberfest but maybe this was your fourth missing PR? 🤞 I think it still counts in that case!


Stocklka feature

Safety feature over 45kph by panda and 12.5m/s in car controller
800 torque limited from car controller and 804 by panda
Future feature may include torque adjustment even when OP is on


ippserver does not pass IPP-1.1 test

Continuing to test ippserver functionality on a Raspberry Pi 3 (running openSUSE vLeap 42.3). Compiled on the Raspi itself from current sources (latest commit was Oct 12, cc48390).
Starting the most simple printer emulator command on the Rasp viai:
 ./server/ippserver -vvvv  "Test Test"

and then running ipptool from a different host:
 ./tools/ipptool -V 1.1 -t  http://leapraspi.local.:8000/ipp/print  examples/ipp-1.1.test

results in this output:
"examples/ipp-1.1.test":
   RFC 2911 section 3.1.1: Bad request-id value 0                       [PASS]
   RFC 2911 section 3.1.4: No Operation Attributes                      [PASS]
   RFC 2911 section 3.1.4: attributes-charset                           [PASS]
   RFC 2911 section 3.1.4: attributes-natural-language                  [PASS]
   RFC 2911 section 3.1.4: attributes-natural-language + attributes-cha [PASS]
   RFC 2911 section 3.1.4: attributes-charset + attributes-natural-lang [PASS]
   RFC 2911 section 3.1.8: Unsupported IPP version 0.0                  [PASS]
   RFC 2911 section 3.2: No printer-uri operation attribute             [PASS]
   RFC 2911 section 3.2.1: Print-Job Operation                          [FAIL]
       RECEIVED: 160 bytes in response
       status-code = client-error-attributes-or-values-not-supported (Unsupported document-format mimeMediaType value.)
       "document-format": Bad mimeMediaType value "" - bad characters (RFC 2911 section 4.1.9).
       EXPECTED: STATUS successful-ok (got client-error-attributes-or-values-not-supported)
       EXPECTED: STATUS client-error-document-format-not-supported (got
       client-error-attributes-or-values-not-supported)
       EXPECTED: STATUS server-error-job-canceled (got client-error-attributes-or-values-not-supported)
       status-message="Unsupported document-format mimeMediaType value."
       EXPECTED: job-uri
       EXPECTED: job-id
       EXPECTED: job-state
       EXPECTED: job-state-reasons

Summary: 9 tests, 8 passed, 1 failed, 0 skipped
Score: 88%

The ipp-1.1.text file I used is from Git sources here, commit 442312b72218d73f84ab02fee351b1dad8df0ad1 (2017, March 28th).

You forgot to specify a print file to use for testing. From the usage at the top of the ipp-1.1.test file:
# Usage:
#
#   ./ipptool -f filename [-d document-uri=SOMEURI] -t printer-uri ipp-1.1.test
#   ./ipptool -f filename -d NOPRINT=1 -t printer-uri ipp-1.1.test
#
# The latter form disables all but the basic file printing.

"You forgot to specify a print file to use for testing."

D'ooooh!
*(Blushing)*


How to create model(.bin, .obj, .gltf or .mtl ) for an .png to use in sceneform Android studio?

If all you wanna do is just show a 2D picture in space you can put that .png inside an ImageView and render the view using ViewRenderable.

That's necessary to add 2d images with shades/ If compulsory them how to get shaders for viewing viewRenders in Android Studion

Can you give more detail on what's being requested and how it's being used?


Implementation vs paper

Hi,
First of all, thanks for this amazing work, it has been a pleasure to dive in the paper !
Now more precisely, when looking at the implementation and the implementation of the actnorm module, I can't understand the choice made considering the paper.
In the paper you state that you used a affine transformation of the activation with parameters s and b.

But in the implementation it seems to first add the bias b: x = x + b(with actnorm_center), then multiply by s: x =  s * (x+b) (with actnorm_scale)
You reverse the code when reverse = True but I feel this might be the opposite.
I surely miss something  as you manage to train the model but I am curious about this choice.
Do I miss something?
Thank you in advance for your help !
Victor

Thanks Victor! :) You're right, we first centre the activations to 0 mean, then scale them to unit variance, we'll update the paper accordingly.
In terms of learning though the formulations are equivalent, since x = s * (x + b) = s * x + (s * b) = s * x + b', where b' is the new bias. Note the loss
term sum(log(|s|)) prevents values in s from becoming 0.


Update socket.dart

Hello,
First of all thanks for your great work in this library.
Please, could you make channel parameters more flexible and update de String, String to String, dynamic?
I need this change to use this library in our app. I think is a simple change that not affect negatively to any library user.

@carlosmobile Sure! I think this was a mistake from me. That needed to support JSON-serializable objects so your type definition reflects that better.
Thanks for the your contribution!

Published in 0.4.7 !

Amazing @matehat !!

We need to change in fromsocket too, sorry.
I/flutter (10251): Exception: type '_InternalLinkedHashMap<String, dynamic>' is not a subtype of type 'Map<String, String>'
I/flutter (10251): #0      PhoenixSocket.addChannel (package:phoenix_socket/src/socket.dart:335:21)


fix($location): don't crash if navigating outside the app base

This should fix #11667

Is this resolved by https://github.com/angular/angular.js/commit/9e492c358c19549696577c86c2c61b93f50ab356 and https://github.com/angular/angular.js/commit/92c7ce5becb9b90216c49f50e0f57f5dbeda8eba?

Yes.


Add mode on URL Launcher's .launch() for OS to handle URLs itself.

In previous version, URL launcher would open banking sites in in-app-webview which was creating issue on redirection after payment is done.

LGTM


Using with the controllerAs syntax

I'm using the controllerAs syntax for angular and have tried the following code which is not working
 _thisCtrl = this;
_thisCtrl.todos = {...};
var keys = ['todos']
$ync(this, keys, 'myRoom')

My browser is throwing an error:
sync.js:163 : Uncaught TypeError: scope.$apply is not a function(…)
Any ideas how to resolve this so I can use your amazing package!
Thanks so much

Hello, sorry I am not using Angular anymore so this project becomes unmaintained.
According to this article I think you still need to use $sync with $scope, prepending it with the name you’ve given to ng-controller.

ok, many thanks!


Input validation on blur should not apply to Pristine fields

I'm comparing the behavior I'm experiencing with ReForm to this example from react-ux-form. In the react-ux-form example notice that first name is required but simply clicking into the field and then clicking out to cause a blur doesn't activate the error message. You have to actually "dirty" the field before the error message will appear.
An example use case where this is important is for a popup newsletter form. When the popup initially appears it auto-focuses the email field input. But a user who immediately clicks the popup-dismiss button is briefly shown an input validation error message even when they didn't mean to interact with the form at all.

Hmm, got it.
Could you provide a reproducible example? Just to make sure that we're on the same page with this behaviour.

If we take the signup form demo, I want to validate the email field on blur. So my naive approach would be:
      <Input
        value={form.values.email}
        onChange={ReForm.Helpers.handleChange(form.handleChange(Email))}
        placeholder="Email"
        error=?{form.getFieldError(Field(Email))}
        onBlur={_ =>
          form.getFieldState(Field(Email)) != Pristine ? form.validateField(Field(Email)) : ()}
      />

but that doesn't work because the field seems to always be pristine.
So I'm forced to simply validate on blur unconditionally.
      <Input
        value={form.values.email}
        onChange={ReForm.Helpers.handleChange(form.handleChange(Email))}
        placeholder="Email"
        error=?{form.getFieldError(Field(Email))}
        onBlur={_ => form.validateField(Field(Email))}
      />

Which gives the undesirable behavior of validating the pristine field.


🗑 Cleanup after CRUD Setup

Cleanup Tasks

[x] Remove Timezone
[x] Free up repository of loose strings
Optimize ExtractorService

[x] Uniform structure for parameter
[x] Reuse common extractor
[x] Update extractors to return a single entity or array of entities

Closed by #218


[WIP] qa: run flake8 against tests

Should go after #178, which has a lot of fixes for this already.

Rebase on #178 / #182.


SaschaWillems can you describe exactly how do render to texture ?

I translate your examples to another language and i need know exactly how do render to texture.
I mean algorithm step by step.
And how copy render result to SwapChain texture

Because in your example not simply understand how this done.

Every example renders to a texture level (i.e. image) provided by the swapchain using an image view.
For a start, see the references to the Vulkan functions and structs vkGetSwapchainImagesKHR, VkImageViewCreateInfo, vkCreateImageView for obtaining the swapchain images and getting image views on them, and then to setup the framebuffer see VkFramebufferCreateInfo with vkCreateFramebuffer. You then have to reference the framebuffer in VkRenderPassBeginInfo when doing vkCmdBeginRenderPass.

@Ronniko : Can you please use the Khronos Vulkan forums for asking Vulkan related questions?


Notifications are not working in Windows 10 Fall Creators Update

While working on a czech translation I noticed that the app should show notification before (micro)break but on Windows (at least Windows 10, Fall Creators Update) it's not working. I never got any notification before any break. I have them enabled in settings.

so I've tried again and it seems to work now. weird, but I guess good ;]


Process could not be started [The system cannot find the path specified. ]

Am Using laravel
env
MAIL_DRIVER=smtp
MAIL_HOST=smtp.gmail.com
MAIL_PORT=465
MAIL_USERNAME=******
MAIL_PASSWORD=********
MAIL_ENCRYPTION=tls
controller
public function sendEmail(Request $request){
Mail::to('chuksdsilent@gmail.com')->send(new Mailtrap());
return response()->json(['message' => 'Request completed']);
}
Swift_TransportException: Process could not be started [The system cannot find the path specified.
] in file C:\xampp\htdocs\tranxavApi\vendor\swiftmailer\swiftmailer\lib\classes\Swift\Transport\StreamBuffer.php on line 299
Stack trace:

Swift_TransportException->() C:\xampp\htdocs\tranxavApi\vendor\swiftmailer\swiftmailer\lib\classes\Swift\Transport\StreamBuffer.php:299
Swift_Transport_StreamBuffer->establishProcessConnection() C:\xampp\htdocs\tranxavApi\vendor\swiftmailer\swiftmailer\lib\classes\Swift\Transport\StreamBuffer.php:58
Swift_Transport_StreamBuffer->initialize() C:\xampp\htdocs\tranxavApi\vendor\swiftmailer\swiftmailer\lib\classes\Swift\Transport\AbstractSmtpTransport.php:126
Swift_Transport_AbstractSmtpTransport->start() C:\xampp\htdocs\tranxavApi\vendor\swiftmailer\swiftmailer\lib\classes\Swift\Transport\SendmailTransport.php:52
Swift_Transport_SendmailTransport->start() C:\xampp\htdocs\tranxavApi\vendor\swiftmailer\swiftmailer\lib\classes\Swift\Mailer.php:67
Swift_Mailer->send() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Mail\Mailer.php:451
Illuminate\Mail\Mailer->sendSwiftMessage() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Mail\Mailer.php:235
Illuminate\Mail\Mailer->send() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Support\Facades\Facade.php:221
Illuminate\Support\Facades\Facade->__callStatic() C:\xampp\htdocs\tranxavApi\app\Http\Controllers\APIController.php:120
App\Http\Controllers\APIController->sendEmail() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Controller.php:54
call_user_func_array() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Controller.php:54
Illuminate\Routing\Controller->callAction() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\ControllerDispatcher.php:45
Illuminate\Routing\ControllerDispatcher->dispatch() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Route.php:212
Illuminate\Routing\Route->runController() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Route.php:169
Illuminate\Routing\Route->run() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Router.php:658
Illuminate\Routing\Router->Illuminate\Routing{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php:30
Illuminate\Routing\Pipeline->Illuminate\Routing{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Middleware\SubstituteBindings.php:41
Illuminate\Routing\Middleware\SubstituteBindings->handle() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php:149
Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php:53
Illuminate\Routing\Pipeline->Illuminate\Routing{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Middleware\ThrottleRequests.php:57
Illuminate\Routing\Middleware\ThrottleRequests->handle() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php:149
Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php:53
Illuminate\Routing\Pipeline->Illuminate\Routing{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php:102
Illuminate\Pipeline\Pipeline->then() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Router.php:660
Illuminate\Routing\Router->runRouteWithinStack() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Router.php:635
Illuminate\Routing\Router->runRoute() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Router.php:601
Illuminate\Routing\Router->dispatchToRoute() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Router.php:590
Illuminate\Routing\Router->dispatch() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Foundation\Http\Kernel.php:176
Illuminate\Foundation\Http\Kernel->Illuminate\Foundation\Http{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php:30
Illuminate\Routing\Pipeline->Illuminate\Routing{closure}() C:\xampp\htdocs\tranxavApi\vendor\fideloper\proxy\src\TrustProxies.php:56
Fideloper\Proxy\TrustProxies->handle() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php:149
Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php:53
Illuminate\Routing\Pipeline->Illuminate\Routing{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Foundation\Http\Middleware\TransformsRequest.php:30
Illuminate\Foundation\Http\Middleware\TransformsRequest->handle() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php:149
Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php:53
Illuminate\Routing\Pipeline->Illuminate\Routing{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Foundation\Http\Middleware\TransformsRequest.php:30
Illuminate\Foundation\Http\Middleware\TransformsRequest->handle() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php:149
Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php:53
Illuminate\Routing\Pipeline->Illuminate\Routing{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Foundation\Http\Middleware\ValidatePostSize.php:27
Illuminate\Foundation\Http\Middleware\ValidatePostSize->handle() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php:149
Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php:53
Illuminate\Routing\Pipeline->Illuminate\Routing{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Foundation\Http\Middleware\CheckForMaintenanceMode.php:46
Illuminate\Foundation\Http\Middleware\CheckForMaintenanceMode->handle() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php:149
Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Routing\Pipeline.php:53
Illuminate\Routing\Pipeline->Illuminate\Routing{closure}() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Pipeline\Pipeline.php:102
Illuminate\Pipeline\Pipeline->then() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Foundation\Http\Kernel.php:151
Illuminate\Foundation\Http\Kernel->sendRequestThroughRouter() C:\xampp\htdocs\tranxavApi\vendor\laravel\framework\src\Illuminate\Foundation\Http\Kernel.php:116
Illuminate\Foundation\Http\Kernel->handle() C:\xampp\htdocs\tranxavApi\public\index.php:55

Given you are on Windows and it appears that the default command is possibly '/usr/sbin/sendmail -bs', then it is likely that the following method is failing because it cannot find sendmail. I'd suggest using the mail or sendmail drivers instead.
https://github.com/swiftmailer/swiftmailer/blob/c63b07e0968134b9bc522c16c521d38be0fabbb3/lib/classes/Swift/Transport/StreamBuffer.php#L283-L301

@tomzx

I'd suggest using the mail or log drivers.

This not working too. But yeah looks like this problem because of windows.

Hi guys... Any workaround for this problem? Same problem here. Thnx

Same error here! Any suggestions to fix this?

I had this same problem and I couldn't find the reason since native php
mail function was working fine. So, I made a new component for my Yii2
project and removed SwiftMailer (at least till we have some senseful
information or a fix here). 👍🏼
Em ter, 21 de ago de 2018 03:20, Tobias Gräber<EMAIL_ADDRESS>escreveu:

check value of $command.
Make sure the path provided in there is correct. should work then.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/swiftmailer/swiftmailer/issues/1080#issuecomment-414562595,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAZLStbPzu8pGwLdoZ-Ua8U38hMjNFBWks5uS6argaJpZM4TTUzc
.

Hi,
I have solved this issue by updating my windows.

same problem laravel 5.7

Same Error???

i'm fetching same error in my website
www.codepopular.com

i have the same problem.Any solution? please

same error

I think the solution here is to update swiftmailer to version 6 or higher but I can't find instructions on how to do that.

any solution??

same error...
any solution?

whats error ??
On Fri, Jul 31, 2020 at 12:16 PM Jayen Patel<EMAIL_ADDRESS>wrote:

same error...
any solution?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/swiftmailer/swiftmailer/issues/1080#issuecomment-666959726,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALDP3T4M2E6YDWEBBA4QNSLR6JSFNANCNFSM4E2NJTOA
.

This is the php mailer issue. when i use smtp credential in localhost then it gave me error.


Concerns with filesystem/block device in the same SD card

I am evaluating mbed cloud client 1.3.3 on NUMAKER_PFM_NUC472/NUMAKER_PFM_M487 and send the same issue as https://github.com/ARMmbed/mbed-cloud-client-example/issues/10. In this case, my SD card is not partitioned (no MBR) but it is used for both FAT file system (to save KCM data) and block device (to save F/W update candidates). My concerns are:

There's potential risk that FAT file system and block device would overlap. Do you have plan to fix it?
FAT file system and block device both would call the underlying block device init/deinit? How do you coordinate them? Because deinit called by one end will make the other end to fail to work with the underlying block device.

I have not personally experienced it but we have had reports of weird behaviour of fat file system infringing on the update storage.
You can use https://github.com/ARMmbed/mbed-os/blob/master/features/filesystem/bd/SlicingBlockDevice.h to create two virtual blockdevices, one for fs one for update storage, so that there is absolutely no chance of them overwriting each other. And init and deinit wouldn't be an issue anymore because it is a virtual device.
Note if you are using virtual blockdevices, you need to be careful regards to the offset of update storage you set. https://github.com/ARMmbed/mbed-cloud-client-example/blob/master/mbed_app.json#L16

init and deinit wouldn't be an issue anymore because it is a virtual device.

Checking SlicingBlockDevice init/deinit, their init/deinit would call _bd init/deinit, where _bd is the underlying block device, e.g. SDBlockDevice. So init/deinit conflict between fs SlicingBlockDevice and update SlicingBlockDevice still exists.
int SlicingBlockDevice::init()
{
    int err = _bd->init();

int SlicingBlockDevice::deinit()
{
    return _bd->deinit();

@LiyouZhou

@ccli8 good point. There are 2 mitigations:

I think we don't call deinit from within the mbed-cloud-client library.
Init should be able to be called multiple times without side effects and the API is stateless. Of course there is no garantee but so far we have not seen problems.

There as been issues with the FlashIAP before https://github.com/ARMmbed/mbed-os/issues/4967. So we will have to fix it in the driver in mbed-os if problem arises again.

@LiyouZhou How about define a rule for block device, for example:

init should be able to be called multiple times without side effect.
Utility block device (ChainingBlockDevice/SlicingBlockDevice/MBRBlockDevice) cannot call deinit of the underlying block device in their deinit to keep sharable not broken.

Then do you have a plan to apply the e.g. SlicingBlockDevice idea to fix the overlap issue?

init should be able to be called multiple times without side effect.
Utility block device (ChainingBlockDevice/SlicingBlockDevice/MBRBlockDevice) cannot call deinit of the underlying block device in their deinit to keep sharable not broken.

These two will probably be down to mbed-so API.

Then do you have a plan to apply the e.g. SlicingBlockDevice idea to fix the overlap issue?
I will create an issue internally.

I raise https://github.com/ARMmbed/mbed-os/issues/7455 to mbed-os to address the issue.

@ccli8 thanks for raising #7455.
There is a PR merged to fix this: https://github.com/ARMmbed/mbed-os/issues/7455

@MarceloSalazar So in next release of Simple Cloud Client template application, you will also fix the overlap issue?

Hi @ccli8 , I understand the overlap issue will need to be fixed in Mbed OS. If there isn't a ticket, please raise one!
The SMCC template application will be updated to newer versions of Mbed OS (with corresponding fixes), but you can also update the mbed-os.lib of your application earlier if needed.


Implement a method in SubarrayDescription to detect array layout name

similar to the camera and optics lookup tables, just need a method to
turn an array layout into a standard array name. Probably we can just
support Prod4 layouts? or even Prod3 if we want to be complete.

We should not introduce more guessing than we already have.
We should implement ways of setting it. Not guessing it from random input data files.

We now assign array name via configuration or from simtel metadata, closing


canvas double click fired on header

When you double click the header dates in the top bar, the onCanvasDoubleClick event fires and reports that the group is the first row.
See the demo example here:
https://codesandbox.io/s/xjljp86z2q
The part (b) of this is that the docs describe the callback signature as onCanvasDoubleClick(groupId, time, e), however as seen in the example above, the groupId is actually the entire group object.

I just updated the readme to accurately portray what's returned.  Eventually I'll get around to updating all callbacks to receive the full object, not just the id

pushed in v0.15.9


SVGs were exported incorrectly

Hi,
Please look at these two example SVG's from the repo. SVG is meant to be scalable yet when zoomed in anti-aliasing is clear. The SVG's are currently are not usable.

That's because of filters. Filters are designed to work on raster images, so SVG is flattened before filters are applied.
Flat and High Contrast versions of icons don't include filters, so they are smaller, faster and scale better.

Well that’s great and all but high contrast is for very different use cases and flat is well… flat. One of the points of these emojis is the 3D look.

Thanks @lloydjatkinson - where are you viewing the SVGs and seeing these issues?

Hi, although almost a year old issue, I do observe this anti-aliasing issue in almost all curved SVG paths. I'm currently viewing the first emoji in the asset list, 1st_place_medal_color.svg via the Microsoft Edge browser and I can see the jagged edges on the path.
For reference:

Hope this provides some context to this issue.


ObjectAssign does not work with nil values

{
    ...{
        foo: "foo",
        bar: "bar",
    },
    foo: undefined,
}

Playground
I would expect something like this to evaluate down to {bar: "bar"}, effectively deleting foo from the object, but because it desugars the remaining keys into an table it looses that the foo key ever exists, so it just inherits it's value from the spread object.

This may be possible to implement by replacing nil with some placeholder value, and having __TS__ObjectAssign detect this value.
We could make explicit fields in object literals a (likely worthwhile) exception to lua/js behavior for nil values in tables.
However, if we decide to do this, how should these other related cases be treated:
const value: unknown = undefined
const obj = {
    ...{ foo: "foo" },
    foo: value
} // erases foo value in js, or sets value to "undefined"

const obj = {
    foo: "foo",
    ...{ foo: undefined }
}
// similar behavior to above

I think its practical to match js behavior in the first case, but ignore the second case as that might be impossible to implement.


Update pyserial dependency to install it with rosdep

This allows rosdep to find the correct serial package and install it.

Thanks for the update!


brandon/milestone report fixes

src/pages/MilestoneReport/MilestoneReport.ts
None - a9d1a3:

Reordered the imports of createRoot and Root in the file.
Updated the getMilestoneData method to include detailed documentation with JSDoc comments.
Refactored the getMilestoneData method to sort achievements before filtering them based on certain criteria.
Added a comment to clarify the filtering logic for achievements in the getMilestoneData method.
src/pages/MilestoneReport/components/MilestoneReport.tsx

None - 49d9a7:

Fixed a typo in the class name from "MilestoneReprtTable" to "MilestoneReportTable".
Added new columns "Next Milestone" and "Complete" to the table.
Renamed the field "personalGrowth" to "personal_growth" for consistency.
Updated column widths and headers for better display and clarity in the table.
Adjusted the column settings in the GridComponent for consistency with the table changes.
src/pages/MilestoneReport/models/MilestonePlanningItem.ts

None - 91f6a9:

Added TerrainAchievementsEventCount to the import statement in the file.
Added a new interface MilestoneObject and classes MilestoneRequirements and MilestoneCurrent implementing MilestoneObject.
Added functions calculateMilestone, calculateEventResult, and calculateOverallPercent.
Modified the class MilestonePlanningItem by adding a new property overall_percent.
Renamed the property personalGrowth to personal_growth in the MilestonePlanningItem class.
Updated the constructor of MilestonePlanningItem to calculate values based on MilestoneRequirements and MilestoneCurrent.
Removed redundant code related to calculating maxL, maxA, maxP, and eventCount in the MilestonePlanningItem constructor.

@bwhite001 let me know if you are ok for me to delete the branch now it is accepted and if we want to do this as the standard process.


Updates to RackHD/test to cleanup module pathing.

Standardized how RackHD/test finds test and test/common.  This
eliminates the need for inlined 'git-rev-parse' style statements
throughout the code.  Doing the following in an individual test
will set up the module pathing:
import fit_path

Just messing around with hound.  Don't merge.


Odd HAL handling of EnsEMBL MAFs

Dear developer,
I encountered a confusing behaviour of HAL with Ensembl MAF files
see attached maf file
very_small.maf.txt
If I run  maf2hal very_small.maf small.hal
everything seems fine
but if I want to extract maf again, I get something very different:
I would expect (from very_small.maf):
a
s homo_sapiens.2                     629356   21392 +  242193529 GCAGAATTTAAAATA
s pan_troglodytes.2A                 619768   21180 +  113622374 GCAGAATTTAAAATA
s gorilla_gorilla.2a                 628232   22420 +  111351968 GCAGAATTTAAAATA
s pongo_abelii.2a                    626026   26257 -  113028656 GCAGAATTTAAAATA
s chlorocebus_sabeus.14              572335   21694 -  107702431 GCAGAATTTAAAATA
s papio_anubis.13                    560811   21646 +  128036923 GCAGAATTTAAAATA
s callithrix_jacchus.14              678752   17020 -  108792865 GCAGAATTTAAAATA
but I get
hal2maf --refGenome homo_sapiens --refSequence 2 --start 629356 --length 16 small.hal stdout
a
s       homo_sapiens.2  629356  16      +       242193529       GCAGAATTTAAAATAA
s       callithrix_jacchus.14   695717  16      -       108792865       TCTTCTGCTCAAAGTC
s       chlorocebus_sabeus.14   593974  16      -       107702431       TCTTCTGCTTGAAGCC
s       gorilla_gorilla.2a      628232  16      +       111351968       GCAGAATTTAAAATAA
s       pan_troglodytes.2A      619768  16      +       113622374       GCAGAATTTAAAATAA
s       papio_anubis.13 560811  16      +       128036923       GCAGAATTTAAAATAA
s       pongo_abelii.2a 652228  16      -       113028656       TCTTCTGCTTGAAGCC
Any ideas ?

Looking like that, it would seem there's a bug in maf2hal's support for
reverse strand blocks.  I'll try to look into it when I have a chance...
On Mon, Aug 29, 2016 at 3:54 AM, CDieterich<EMAIL_ADDRESS>wrote:

Dear developer,
I encountered a confusing behaviour of HAL with Ensembl MAF files
see attached maf file
very_small.maf.txt
https://github.com/glennhickey/hal/files/441920/very_small.maf.txt
If I run maf2hal very_small.maf small.hal
everything seems fine
but if I want to extract maf again, I get something very different:
I would expect (from very_small.maf):
a
s homo_sapiens.2<PHONE_NUMBER>2 + 242193529 GCAGAATTTAAAATA
s pan_troglodytes.2A<PHONE_NUMBER>0 + 113622374 GCAGAATTTAAAATA
s gorilla_gorilla.2a<PHONE_NUMBER>0 + 111351968 GCAGAATTTAAAATA
s pongo_abelii.2a<PHONE_NUMBER>7 - 113028656 GCAGAATTTAAAATA
s chlorocebus_sabeus.14<PHONE_NUMBER>4 - 107702431 GCAGAATTTAAAATA
s papio_anubis.13<PHONE_NUMBER>6 + 128036923 GCAGAATTTAAAATA
s callithrix_jacchus.14<PHONE_NUMBER>0 - 108792865 GCAGAATTTAAAATA
but I get
hal2maf --refGenome homo_sapiens --refSequence 2 --start 629356 --length
16 small.hal stdout
a
s homo_sapiens.2 629356 16 + 242193529 GCAGAATTTAAAATAA
s callithrix_jacchus.14 695717 16 - 108792865 TCTTCTGCTCAAAGTC
s chlorocebus_sabeus.14 593974 16 - 107702431 TCTTCTGCTTGAAGCC
s gorilla_gorilla.2a 628232 16 + 111351968 GCAGAATTTAAAATAA
s pan_troglodytes.2A 619768 16 + 113622374 GCAGAATTTAAAATAA
s papio_anubis.13 560811 16 + 128036923 GCAGAATTTAAAATAA
s pongo_abelii.2a 652228 16 - 113028656 TCTTCTGCTTGAAGCC
Any ideas ?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/glennhickey/hal/issues/50, or mute the thread
https://github.com/notifications/unsubscribe-auth/AA2_7uiFWhH7ZKFLJ0tjOcZ674hku8qbks5qkpA8gaJpZM4JvOw3
.

Ok, thanks.
Would you know of any substitute in the meantime.
UCSC mafs should or should not work ?
Cheers

The thing is, we generally output HAL directly with Cactus without ever
running maf2hal.  So I'm worried maf2hal got broken without being noticed,
and I'm not sure any MAF is going to work.  Will try to verify this today.
On Wed, Aug 31, 2016 at 1:54 AM, CDieterich<EMAIL_ADDRESS>wrote:

Ok, thanks.
Would you know of any substitute in the meantime.
UCSC mafs should or should not work ?
Cheers
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/glennhickey/hal/issues/50#issuecomment-243665714, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA2_7loD8ycHFkErw26aTbnSco8I3twnks5qlRcKgaJpZM4JvOw3
.

Oh I see. Thank you for looking into this.
My main motivation for using HAL is to get fast and seamless
access to whole genome alignments of primates and vertebrates.
However, I do not have the compute capacity to compute MAF/Cactus alignments from scratch.

OK, this should hopefully be fixed now.
Thanks for your patience, and for the clear example!
On Thu, Sep 1, 2016 at 1:50 AM, CDieterich<EMAIL_ADDRESS>wrote:

Oh I see. Thank you for looking into this.
My main motivation for using HAL is to get fast and seamless
access to whole genome alignments of primates and vertebrates.
However, I do not have the compute capacity to compute MAF/Cactus
alignments from scratch.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/glennhickey/hal/issues/50#issuecomment-243982596, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA2_7nERuzQMIMF9mrUPzx_tyBjiXCNbks5qlmeagaJpZM4JvOw3
.

Ok, after running
mafStrander --maf pruned.maf --seq homo_sapiens --strand + > positive.maf
to put the reference species on the '+' strand. Everything seems to work.
Thank you !


Launching batch extractors on Roger

@robkooper @yanliu-chn @jterstriep follow up from meeting today. Goal is to get all TERRA data from the summer processed through all extractors.
Yan will update the nco module on Roger to new version.
Max will get dependencies installed on Roger and write launcher.sh for each extractor that will load virtualenv, modules, etc. and runs extractor. This will start one instance. Rob can help if needed.
Jeff will write qsub wrapper to call the launcher.sh and start many instances of extractors.
We can discuss here.

@jterstriep here is the HPC launcher script for plantcv extractor, for your reference: https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-plantcv/browse

@jterstriep  I have installed dependencies for demosaic extractor on Roger head node as a first extractor.
(@robkooper @jdmaloney @yanliu-chn FYI)

started in /home/mburnet2/extractors
created a python virtualenv called pyenv
activated it & installed dependencies & cloned+installed pyClowder
git clone the extractor repo
update extractor config.py for our terra Clowder instance
script that should launch the extractor: https://github.com/terraref/extractors-stereo-rgb/blob/master/demosaic/batch_launcher.sh

...not sure if having the files in my home directory will cause permissions problems.
the extractor started running but failed quickly with "file name too long" error:
OUT FILE NAME:
/projects/arpae/terraref/sites/ua-mac/Level_1/demosaic/2016-10-30/2016-10-30__10-48-08-731/33a5f544-681c-42b6-adb1-fd09ad8c4c99_right.tif
ERROR 1: TIFFOpen:????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????f罻: File name too long
OK

The JPGs were generated properly, but this file path is apparently too long for GDAL. On my VM elsewhere, I didn't encounter that error but filenames were slightly shorter:
OK ON VM:
/home/extractor/sites/ua-mac/Level_1/demosaic/2016-10-30/2016-10-30__10-36-05-536/88867503-f576-47de-926e-dee347007c5c_left.tif

TOO LONG ON ROGER:
/projects/arpae/terraref/sites/ua-mac/Level_1/demosaic/2016-10-30/2016-10-30__10-35-35-433/a41f1694-3e6f-4ae9-b379-bb6d41151005_left.tif

I'm wondering if we'll need to use some kind of symlinks or something to pass GDAL a shorter filepath for the output TIFs. The JPGs are theoretically the same file length and cause no errors, but it looks like those are piped through numpy rather than GDAL for processing. I'm going to experiment a bit more.

@jterstriep @robkooper @yanliu-chn @jdmaloney @dlebauer
OK, updated the code to address the "file name too long issue":

extractor will generate the TIFF files as temp files (e.g. /tmp/tmpZ1VjGH)
temp files are copied over to the destination in Level_1 as full filename
original temp files removed
full paths in Level_1 are sent to clowder

This might be required for other extractors that use GDAL, but most scripts do not have a problem with long file paths.
@jterstriep with that in mind, I think we are ready to test this. we want to run the launcher.sh script as David Lebauer (dlebauer) so the output files will be owned by him, which is how the existing extractor deployments work.
To start an instance we should just need:
cd /home/mburnet2/extractors/extractors-stereo-rgb/demosaic
./batch_launcher.py

This will load required modules, activate python environment, and start listening to RabbitMQ.
Let me know if the files being in my home directory is a problem, but you should have execute permissions I think.

I'll get the other extractors prepped in the meantime (git clone on Roger, install additional dependencies) but we can use this first one as a test case. Note that I have an instance of this extractor running on separate VM at the moment - i will leave it running in the meantime.

@jterstriep also ready:
cd /home/mburnet2/extractors/extractors-multispectral/flir
./batch_launcher.py

@jterstriep I just moved the hyperspectral extractor latest code out there as well:
cd /home/mburnet2/extractors/extractors-hyperspectral/hyperspectral
./batch_launcher.py

...I haven't really been able to test this fully due to the memory requirements, but it seemed to get at least that far in my local testing. Charlie reports here that he was able to get it running successfully:
https://github.com/terraref/computing-pipeline/issues/194
...not sure if it will require a different instantiation approach than the others?

the PLY -> LAS converter for 3d scanner data should also be ready:
cd /home/mburnet2/extractors/extractors-3dscanner/ply2las
./batch_launcher.py

@max-zilla I can't see anything in your home directory. Can you place the code in the project space?

@jterstriep I'll try to do so after the terra call from 10-11.
I think if I understand python virtualenv correctly, I should be able to just move the pyenv directory outside my home and still have it functional (we'll have to update the batch_launcher scripts to point to correct place).
I'll put things here:
/projects/arpae/terraref/shared/extractors

@jterstriep OK, just copied the files over, since it's /shared i hope you can access.
I'll update other scripts soon but I quickly updated one in-place for you:
/projects/arpae/terraref/shared/extractors/extractors-3dscanner/ply2las

batch_launcher.sh has been edited to point to the /projects location instead of /home/mburnet2.

@jterstriep OK, here's something you can use for testing.

we'll use the environmentlogger extractor for initial testing. it's nice because it operates on a file (vs. a dataset of many files) and in normal operation we only see one of these environmentlogger.json files per hour so there won't be new files appearing on the queue very often)
the queue is already empty in RMQ
I shut down the environmentlogger on VM so it won't steal messages
created a TEST dataset + file:

https://terraref.ncsa.illinois.edu/clowder/datasets/581b724b4f0c2495647794a9
https://terraref.ncsa.illinois.edu/clowder/files/581b72da4f0c24956477971b
this file is owned by Maricopa Site - i'll bring you credentials on a post-it

When you submit the file for extraction, it will reprocess and generate a .nc file corresponding to the json file. This is a bit different than the proper pipeline because the test file I uploaded was not already on Roger (for most files we copy to roger then just give Clowder the pointer) and there's already an .nc file for this file on Roger from the proper pipeline - so the extractor won't be able to write the .nc file to that output location, but if it gets that far I think we're OK.
Basically you can

start extractor
go to the file in Clowder, logging in as Maricopa Site
click 'Submit file for extraction' towards the bottom under Extractions
submit to terra.environmentlogger with no parameters

...should kick off extractor. you know it runs successfully if an .nc file shows up in the dataset when you refresh it.

@robkooper FYI the demosaic queue went from ~104,000 last night when i stopped the pipeline to ~36,000 this morning with one extractor running.

I couldn't get the terra.ply2las.py run. I finally tracked down the problem to a bad virtualenv at /projects/arpae/terraref/shared/extractors/pyenv/bin/activate. Once activated, python still points to the ROGER module python rather than the wrapper in the virtualenv so the terra.ply2las.py doesn't find pyclowder.
Should I fix this? Is anything else using /projects/arpae/terraref/shared/extractors/pyenv at this point?

@jterstriep I also realized during testing yesterday that the extractor requires two additional modules to be loaded or it will fail on the actual extraction (GCC and proj4) - I updated the batch_launcher.sh and ran git pull on the one you're using.
You can fix that, sure - nothing else is using that environment. I had copied it from /home/mburnet2/extractors/pyenv rather than recreating it which in retrospect probably screwed up some paths. Thanks.

@max-zilla OK, I'll rebuild it.
We might want to revisit this. I'm beginning to think batch_launcher.sh for each extractor might not be the best approach.

@jterstriep uploaded a sample PLY file to Google Drive for you:
https://drive.google.com/file/d/0B42lleyRvKnlaFlWRXQyMzhCZ1U/view?usp=sharing
Verified at http://rabbitmq.ncsa.illinois.edu:15672/#/queues/clowder/terra.ply2las that the ply2las queue is empty with 0 consumers - uploading that PLY file to a new dataset as the Maricopa Site user in Clowder should trigger a message that your extractor will execute.
One caveat is that the extractor will need to write the resulting output to projects/arpae/terraref/sites/ua-mac/Level_1/scanner3DTop directory so the user running the extractor must have permissions there. @jdmaloney gave my account permission for that to run the two extractions yesterday, but @robkooper mentioned we could also run the extractor as David LeBauer (dlebauer) who owns all those directories.
The extractor will have run successfully if an LAS file appears in the dataset, but if you get python errors that you don't have permission for the Level_1 output directory the extractor basically ran successfully at that point and it's just a permissions issue.

@jterstriep any updates on this?
the terra.demosaic queue is slowly filling up since there's only 1 extractor currently deployed on a VM - up to 257,000 messages. this is this one:
cd /projects/arpae/terraref/shared/extractors/extractors-stereo-rgb/demosaic
./batch_launcher.py

(most of those messages are datasets that will be checked and determined to be irrelevant, but they still need to be checked)
...this might be a good case to test with.

@robkooper @jterstriep
Rob, Jeff has pushed some edits to the PLY->LAS extractor based on making things better suited for the HPC environment. He took a look at how we had the extractors set up on Roger and had some modifications to the code to better prepare for HPC environment.
These are edits to a pyClowder 1 extractor. We went over those changes together and several of them fit the direction we're heading for pyClowder 2. I thought it would make sense to share those changes so we can discuss whether to update pyClowder 2 itself to include aspects of them.
One change is already in pyClowder 2 - getting rid of config.py and making those parameters into command line args. The other change is adding a setup.py script that installs the python script + dependencies as a proper module.
Links:

https://github.com/terraref/extractors-3dscanner/compare/py_package comparison of Jeff's changes to original extractor code
@jterstriep here is one extractor I've updated to pyclowder 2: https://github.com/terraref/extractors-environmental/tree/master/environmentlogger This removes config.py as well but doesn't include config.py.

I think we can get the best of both worlds here by updating the extractors to pyclowder 2 (I was going to work on that anyway) and adding setup.py for each from Jeff's template.
Next day or so I'm going to run the other Mongo queries to generate UUID lists for queuing for extraction manually, and I'll test the manually extraction queuing a bit on datasets to see whether the possible issue I mentioned last week exists (where the RMQ message sent there does not include datasetID we need).

Needed to create pull request for Clowder to support doing this for files: https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/clowder/pull-requests/1085/overview
The dataset extractors (aka the majority of them) should be fine.
POST https://terraref.ncsa.illinois.edu/clowder/api/datasets/123456/extractions
{"extractor": "extractor.name"}

Now to generate the UUID lists.

Creating sub-issues for the final drive to completion.
#212
#213
#214

@jterstriep couple items
https://github.com/terraref/computing-pipeline/issues/213

i'm setting up a VM on Roger for final 2 extractors here that weren't deployed on a VM before, meteorological (datparser) and 3dscanner ply2las. Then I can close this issue.

I updated hyperspectral extractor to pyClowder 2 and installed pyClowder 2 in the pyenv for this installation on Roger:
/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/

The extractor can be launched using:
/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/batch_launcher.sh

I created a couple datasets on clowder-dev that should trigger the extractor:
https://terraref.ncsa.illinois.edu/clowder-dev/datasets/5852d1e3e4b05de61f745488
https://terraref.ncsa.illinois.edu/clowder-dev/datasets/5852d5c9e4b05de61f7455f0

I ran the extractor for a moment, just to register the queue on clowder-dev:
http://rabbitmq.ncsa.illinois.edu:15672/#/queues/clowder-dev/terra.hyperspectral
There's one message there right now, waiting to process. I can trigger more as well.

If we can verify the dataset above is successfully processed, and Clowder production is fixed, we will just need to modify batch_launcher.sh to point at the Production instance instead of the dev instance.

@jterstriep some progress - qsub is starting & triggering hyperspectral extractor correctly, but I found a new wrinkle.
Most of Zender's code is in a file called hyperspectral_workflow.sh in the same directory as the python script. In the extractor python script, it uses subprocess to call:
returncode = subprocess.call(["bash", "hyperspectral_workflow.sh", "-d", "1", "-i", target_files['raw']['path'], "-o", outFilePath])

...now this works fine when calling python directly, but in the batch job context I need to change something:
bash: hyperspectral_workflow.sh: No such file or directory
2016-12-15 14:09:18,335 [Connector-0    ] ERROR   : root - script encountered an error
2016-12-15 14:09:18,335 [Connector-0    ] ERROR   : root - no output file was produced

Not a surprise that wherever the job is executing doesn't know what the heck this shell script is that it refers to. Question is, what do you think best way to handle is?

make --script a cmd line arg where you pass in full qualified path:
--script /projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/hyperspectral_workflow.sh

use qsub -d to define a working directory (https://wikis.nyu.edu/display/NYUHPC/Tutorial+-+Submitting+a+job+using+qsub#Tutorial-Submittingajobusingqsub-work_dir) e.g.

qsub -l nodes=1 -d /projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/ batch_launcher.sh

I tried this initially, but got an email that makes me think I need /gpfs/largeblockFS/ or something similar?:
ROGER Job ID: 50054.cg-gpu01
 Job Name: batch_launcher.sh
 Exec host list suppressed in email
 An error has occurred processing your job, see below.
 Post job file processing error; job 50054.cg-gpu01 on host cg-cmp16

Unable to copy file 50054.cg-gpu01.OU to /gpfs/largeblockFS/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/batch_launcher.sh.o50054, error 1
*** error from copy
/bin/cp: cannot stat `50054.cg-gpu01.OU': No such file or directory
*** end error output

Some other solution better than these?

I like option 1 since that does not change the working directory.

If you qsub a script, it copies to a tmp location anyway.
The location of the workflow shell script should be part of the extractor config. That means you can define an entry in config.py in the extractor. If the location changes from one deployment to another, you can refer to an env var in config.py, while the env var is set up in $HOME/.bashrc
Thanks,
Yan
From: Max Burnette <notifications<EMAIL_ADDRESS>Reply-To: terraref/computing-pipeline <reply<EMAIL_ADDRESS>Date: Thursday, December 15, 2016 at 2:28 PM
To: terraref/computing-pipeline <computing-pipeline<EMAIL_ADDRESS>Cc: Yan Liu <yanliu<EMAIL_ADDRESS>Assign <assign<EMAIL_ADDRESS>Subject: Re: [terraref/computing-pipeline] Launching batch extractors on Roger (#195)
@jterstriephttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jterstriep&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=QbY3HDLn4TqJD-LXnNSvwTwwOferKDqWlH-gZd7YVUQ&m=g9qovujp2K59adAD29v35wJwWqMRwowsxIv05AKuSZ0&s=siscXZ3ovUNTGOAbdtK05YBMI8NxyykM2dpJtReCm1E&e= some progress - qsub is starting & triggering hyperspectral extractor correctly, but I found a new wrinkle.
Most of Zender's code is in a file called hyperspectral_workflow.sh in the same directory as the python script. In the extractor python script, it uses subprocess to call:
returncode = subprocess.call(["bash", "hyperspectral_workflow.sh", "-d", "1", "-i", target_files['raw']['path'], "-o", outFilePath])
...now this works fine when calling python directly, but in the batch job context I need to change something:
bash: hyperspectral_workflow.sh: No such file or directory
2016-12-15 14:09:18,335 [Connector-0    ] ERROR   : root - script encountered an error
2016-12-15 14:09:18,335 [Connector-0    ] ERROR   : root - no output file was produced
Not a surprise that wherever the job is executing doesn't know what the heck this shell script is that it refers to. Question is, what do you think best way to handle is?

make --script a cmd line arg where you pass in full qualified path:
--script /projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/hyperspectral_workflow.sh

use qsub -d to define a working directory (https://wikis.nyu.edu/display/NYUHPC/Tutorial+-+Submitting+a+job+using+qsub#Tutorial-Submittingajobusingqsub-work_dirhttps://urldefense.proofpoint.com/v2/url?u=https-3A__wikis.nyu.edu_display_NYUHPC_Tutorial-2B-2D-2BSubmitting-2Ba-2Bjob-2Busing-2Bqsub-23Tutorial-2DSubmittingajobusingqsub-2Dwork-5Fdir&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=QbY3HDLn4TqJD-LXnNSvwTwwOferKDqWlH-gZd7YVUQ&m=g9qovujp2K59adAD29v35wJwWqMRwowsxIv05AKuSZ0&s=XTqmpvh8RyQRMnTG8A_WOPj9GhxhrsROm7HUST8VMJ0&e=) e.g.

qsub -l nodes=1 -d /projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/ batch_launcher.sh
I tried this initially, but got an email that makes me think I need /gpfs/largeblockFS/ or something similar?:
ROGER Job ID: 50054.cg-gpu01
Job Name: batch_launcher.sh
Exec host list suppressed in email
An error has occurred processing your job, see below.
Post job file processing error; job 50054.cg-gpu01 on host cg-cmp16
Unable to copy file 50054.cg-gpu01.OU to /gpfs/largeblockFS/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/batch_launcher.sh.o50054, error 1
*** error from copy
/bin/cp: cannot stat `50054.cg-gpu01.OU': No such file or directory
*** end error output

Some other solution better than these?

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_terraref_computing-2Dpipeline_issues_195-23issuecomment-2D267435082&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=QbY3HDLn4TqJD-LXnNSvwTwwOferKDqWlH-gZd7YVUQ&m=g9qovujp2K59adAD29v35wJwWqMRwowsxIv05AKuSZ0&s=vHbUy3TMYWnD2FP_6yzYsCADbsGIZ5q8Ijl4HNWvRbw&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AEArrR3bcfpPB-2Dr4-2DwdJObKqwMBUfs1sks5rIaLRgaJpZM4KjvZB&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=QbY3HDLn4TqJD-LXnNSvwTwwOferKDqWlH-gZd7YVUQ&m=g9qovujp2K59adAD29v35wJwWqMRwowsxIv05AKuSZ0&s=YsiFAai_fpvHqXZtAA22Mh26kfOyqJP7DZlpRfv__T0&e=.

I suggest option 3.
Implement proper setup.py files so that when the extractor is installed both the python script and the shell script get copied to the bin directory (in this case,  /gpfs/largeblockFS/projects/arpae/terraref/shared/extractors) and the hyperspectral_workflow.sh script will be found using $PATH. Plus all the other benefits I discussed this morning.

@yanliu-chn, config.py is no more!

The /gpfs/largeblockFS/ stuff is somewhat irrelavant: pbs tries to copy the stdout/stderr to where it should be (by default, the dir where the job was submitted), but somehow it tried to do that after the job is done and the temp stderr/stdout was deleted.
From: Max Burnette <notifications<EMAIL_ADDRESS>Reply-To: terraref/computing-pipeline <reply<EMAIL_ADDRESS>Date: Thursday, December 15, 2016 at 2:28 PM
To: terraref/computing-pipeline <computing-pipeline<EMAIL_ADDRESS>Cc: Yan Liu <yanliu<EMAIL_ADDRESS>Assign <assign<EMAIL_ADDRESS>Subject: Re: [terraref/computing-pipeline] Launching batch extractors on Roger (#195)
@jterstriephttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jterstriep&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=QbY3HDLn4TqJD-LXnNSvwTwwOferKDqWlH-gZd7YVUQ&m=g9qovujp2K59adAD29v35wJwWqMRwowsxIv05AKuSZ0&s=siscXZ3ovUNTGOAbdtK05YBMI8NxyykM2dpJtReCm1E&e= some progress - qsub is starting & triggering hyperspectral extractor correctly, but I found a new wrinkle.
Most of Zender's code is in a file called hyperspectral_workflow.sh in the same directory as the python script. In the extractor python script, it uses subprocess to call:
returncode = subprocess.call(["bash", "hyperspectral_workflow.sh", "-d", "1", "-i", target_files['raw']['path'], "-o", outFilePath])
...now this works fine when calling python directly, but in the batch job context I need to change something:
bash: hyperspectral_workflow.sh: No such file or directory
2016-12-15 14:09:18,335 [Connector-0    ] ERROR   : root - script encountered an error
2016-12-15 14:09:18,335 [Connector-0    ] ERROR   : root - no output file was produced
Not a surprise that wherever the job is executing doesn't know what the heck this shell script is that it refers to. Question is, what do you think best way to handle is?

make --script a cmd line arg where you pass in full qualified path:
--script /projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/hyperspectral_workflow.sh

use qsub -d to define a working directory (https://wikis.nyu.edu/display/NYUHPC/Tutorial+-+Submitting+a+job+using+qsub#Tutorial-Submittingajobusingqsub-work_dirhttps://urldefense.proofpoint.com/v2/url?u=https-3A__wikis.nyu.edu_display_NYUHPC_Tutorial-2B-2D-2BSubmitting-2Ba-2Bjob-2Busing-2Bqsub-23Tutorial-2DSubmittingajobusingqsub-2Dwork-5Fdir&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=QbY3HDLn4TqJD-LXnNSvwTwwOferKDqWlH-gZd7YVUQ&m=g9qovujp2K59adAD29v35wJwWqMRwowsxIv05AKuSZ0&s=XTqmpvh8RyQRMnTG8A_WOPj9GhxhrsROm7HUST8VMJ0&e=) e.g.

qsub -l nodes=1 -d /projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/ batch_launcher.sh
I tried this initially, but got an email that makes me think I need /gpfs/largeblockFS/ or something similar?:
ROGER Job ID: 50054.cg-gpu01
Job Name: batch_launcher.sh
Exec host list suppressed in email
An error has occurred processing your job, see below.
Post job file processing error; job 50054.cg-gpu01 on host cg-cmp16
Unable to copy file 50054.cg-gpu01.OU to /gpfs/largeblockFS/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/batch_launcher.sh.o50054, error 1
*** error from copy
/bin/cp: cannot stat `50054.cg-gpu01.OU': No such file or directory
*** end error output

Some other solution better than these?

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_terraref_computing-2Dpipeline_issues_195-23issuecomment-2D267435082&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=QbY3HDLn4TqJD-LXnNSvwTwwOferKDqWlH-gZd7YVUQ&m=g9qovujp2K59adAD29v35wJwWqMRwowsxIv05AKuSZ0&s=vHbUy3TMYWnD2FP_6yzYsCADbsGIZ5q8Ijl4HNWvRbw&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AEArrR3bcfpPB-2Dr4-2DwdJObKqwMBUfs1sks5rIaLRgaJpZM4KjvZB&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=QbY3HDLn4TqJD-LXnNSvwTwwOferKDqWlH-gZd7YVUQ&m=g9qovujp2K59adAD29v35wJwWqMRwowsxIv05AKuSZ0&s=YsiFAai_fpvHqXZtAA22Mh26kfOyqJP7DZlpRfv__T0&e=.

Thanks all. For this afternoon I'll just quickly add a command line arg, and if it works I'll later use your 3dscanner setup.py as a template to update as you suggest @jterstriep .

Ok. Then we can assume there is always an env var $EXTRACTOR_WORKFLOW_BINDIR there. Then in python extractor code, send $EXTRACTOR_WORKFLOW_BINDIR/ex.sh to bash.
From: Jeff Terstriep <notifications<EMAIL_ADDRESS>Reply-To: terraref/computing-pipeline <reply<EMAIL_ADDRESS>Date: Thursday, December 15, 2016 at 3:21 PM
To: terraref/computing-pipeline <computing-pipeline<EMAIL_ADDRESS>Cc: Yan Liu <yanliu<EMAIL_ADDRESS>Mention <mention<EMAIL_ADDRESS>Subject: Re: [terraref/computing-pipeline] Launching batch extractors on Roger (#195)
@yanliu-chnhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_yanliu-2Dchn&d=DQMCaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=QbY3HDLn4TqJD-LXnNSvwTwwOferKDqWlH-gZd7YVUQ&m=BPRvCI41aRvkI1PpjyRS6V6vDOb9VDzU8x7p3bR2PGE&s=PCOxxduVhEkMzb8sob6gpTaPBxE0wxAlaHCHAxvWY04&e=, config.py is no more!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_terraref_computing-2Dpipeline_issues_195-23issuecomment-2D267447430&d=DQMCaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=QbY3HDLn4TqJD-LXnNSvwTwwOferKDqWlH-gZd7YVUQ&m=BPRvCI41aRvkI1PpjyRS6V6vDOb9VDzU8x7p3bR2PGE&s=DT384bPLKXyqa2R79N96FY7E9u_YJEnXsKHzavzFWFE&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AEArrQWAUAVQMDvTlFnZNHb9-5Fy-2D5-5FGEmks5rIa9hgaJpZM4KjvZB&d=DQMCaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=QbY3HDLn4TqJD-LXnNSvwTwwOferKDqWlH-gZd7YVUQ&m=BPRvCI41aRvkI1PpjyRS6V6vDOb9VDzU8x7p3bR2PGE&s=SbTwlaLHQb-ePcPa5sQYG5BXVWRnd9O_K23mG8hdwuY&e=.

@czender i just ran the hyperspectral extractor on Roger with this dataset:
https://terraref.ncsa.illinois.edu/clowder/datasets/587024414f0c0dbad1a78b83
I duplicated this dataset on clowder-dev to make sure the extractor would work even if files weren't in a specific folder structure on ROGER, i.e. in an arbitrary Clowder instance, so I updated my code to move any /tmp files into one directory so your _workflow script can be called.
I got an error on the test, although it seems like the processing was done correctly. Here's some output:
testFrameTimeHasCorrectCalendarAttr (__main__.HyperspectralWorkflowTest) ... ok
testFrameTimeHasCorrectUnitsAttr (__main__.HyperspectralWorkflowTest) ... ok
testFrameTimeHasCorrectValue (__main__.HyperspectralWorkflowTest) ... ok
testHistoryIsCorrectlyRecorded (__main__.HyperspectralWorkflowTest) ... ok
testTheNumberOfDimensionsInRootLevelIsCorrect (__main__.HyperspectralWorkflowTest) ... expected failure
testTheWavelengthDimensionsHaveCorrectValues (__main__.HyperspectralWorkflowTest) ... ok
testTheXDimensionsHaveCorrectValues (__main__.HyperspectralWorkflowTest) ... ok
testTheYDimensionsMatchesTimeDimension (__main__.HyperspectralWorkflowTest) ... ok
testWavelengthArrayHasCorrectData (__main__.HyperspectralWorkflowTest) ... ok
testWavelengthArrayHasEnoughData (__main__.HyperspectralWorkflowTest) ... ok
testXHasEnoughAttributes (__main__.HyperspectralWorkflowTest) ... ok
testYHasEnoughAttributes (__main__.HyperspectralWorkflowTest) ... ok
testYHaveCorrectValuesAndAttributes (__main__.HyperspectralWorkflowTest) ... FAIL

======================================================================
FAIL: testYHaveCorrectValuesAndAttributes (__main__.HyperspectralWorkflowTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/largeblockFS/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/hyperspectral_test.py", line 227, in testYHaveCorrectValuesAndAttributes
    self.assertEqual(len(self.y), 169,  msg="The height of the image should always be 169 pxl")
AssertionError: The height of the image should always be 169 pxl

----------------------------------------------------------------------
Ran 13 tests in 0.039s

FAILED (failures=1, expected failures=1)
2017-01-10 13:11:25,375 [Connector-0    ] INFO    : root - uploading /projects/arpae/terraref/sites/ua-mac/Level_1/hyperspectral_manualcheck/2016-12-11/2016-12-11__14-17-02-040/52079867-350a-48a6-9ef3-2f6049cc7b0f_.nc

And your script output:
Terraref hyperspectral data workflow invoked with:
hyperspectral_workflow.sh -d 1 -i /tmp/tmpwhg88m/52079867-350a-48a6-9ef3-2f6049cc7b0f_raw -o /projects/arpae/terraref/sites/ua-mac/Level_1/hyperspectral_manualcheck/2016-12-11/2016-12-11__14-17-02-040/52079867-350a-48a6-9ef3-2f6049cc7b0f_.nc
Hyperspectral workflow scripts in directory /gpfs/largeblockFS/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral
NCO version "4.6.2-beta03" from directory /gpfs/smallblockFS/sw/nco-4.6.2-beta03/bin
Intermediate/temporary files written to directory /gpfs_scratch/arpae/imaging_spectrometer
Final output stored in directory /projects/arpae/terraref/sites/ua-mac/Level_1/hyperspectral_manualcheck/2016-12-11/2016-12-11__14-17-02-040
Input #00: /tmp/tmpwhg88m/52079867-350a-48a6-9ef3-2f6049cc7b0f_raw
trn(in)  : /tmp/tmpwhg88m/52079867-350a-48a6-9ef3-2f6049cc7b0f_raw
trn(out) : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_trn.nc.pid113572.fl00.tmp
ncks -O --hdr_pad=10000 --no_tmp_fl --trr_wxy=955,1600,135 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/tmp/tmpwhg88m/52079867-350a-48a6-9ef3-2f6049cc7b0f_raw /gpfs/largeblockFS/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/hyperspectral_dummy.nc /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_trn.nc.pid113572.fl00.tmp
att(in)  : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_trn.nc.pid113572.fl00.tmp
att(out) : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid113572.fl00.tmp
ncatted -O --gaa terraref_script=hyperspectral_workflow.sh --gaa terraref_hostname=cg-cmp15 --gaa terraref_version="4.6.2-beta03" -a "Conventions,global,o,c,CF-1.5" -a "Project,global,o,c,TERRAREF" --gaa history="Tue Jan 10 13:10:43 CST 2017: hyperspectral_workflow.sh -d 1 -i /tmp/tmpwhg88m/52079867-350a-48a6-9ef3-2f6049cc7b0f_raw -o /projects/arpae/terraref/sites/ua-mac/Level_1/hyperspectral_manualcheck/2016-12-11/2016-12-11__14-17-02-040/52079867-350a-48a6-9ef3-2f6049cc7b0f_.nc" /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_trn.nc.pid113572.fl00.tmp /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid113572.fl00.tmp
jsn(in)  : /tmp/tmpwhg88m/52079867-350a-48a6-9ef3-2f6049cc7b0f_raw
jsn(out) : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_jsn.nc.pid113572
python /gpfs/largeblockFS/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/hyperspectral_metadata.py dbg=yes fmt=4 ftn=no /tmp/tmpwhg88m/52079867-350a-48a6-9ef3-2f6049cc7b0f_raw /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_jsn.nc.pid113572.fl00.tmp
mrg(in)  : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_jsn.nc.pid113572.fl00.tmp
mrg(out) : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid113572.fl00.tmp
ncks -A /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_jsn.nc.pid113572.fl00.tmp /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid113572.fl00.tmp
mrg(in)  : /gpfs/largeblockFS/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/calibration_vnir_25ms.nc
mrg(out) : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid113572.fl00.tmp
ncks -A -C -v xps_img_wht,xps_img_drk /gpfs/largeblockFS/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/calibration_vnir_25ms.nc /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid113572.fl00.tmp
clb(in)  : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid113572.fl00.tmp
clb(out) : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_clb.nc.pid113572.fl00.tmp
ncap2 -A --hdr_pad=10000 -s @drc_spt='"/gpfs/largeblockFS/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral"' -S /gpfs/largeblockFS/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/hyperspectral_calibration.nco /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid113572.fl00.tmp /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid113572.fl00.tmp
Setting parser(filename)=/gpfs/largeblockFS/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/hyperspectral_calibration.nco
/bin/mv -f /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid113572.fl00.tmp /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_clb.nc.pid113572.fl00.tmp
rip(in)  : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_clb.nc.pid113572.fl00.tmp
rip(out) : /projects/arpae/terraref/sites/ua-mac/Level_1/hyperspectral_manualcheck/2016-12-11/2016-12-11__14-17-02-040/52079867-350a-48a6-9ef3-2f6049cc7b0f_.nc
/bin/mv -f /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_clb.nc.pid113572.fl00.tmp /projects/arpae/terraref/sites/ua-mac/Level_1/hyperspectral_manualcheck/2016-12-11/2016-12-11__14-17-02-040/52079867-350a-48a6-9ef3-2f6049cc7b0f_.nc
Cleaning-up intermediate files...
QA/QC check found with 1 or more unexpected FAILURES
Quick views of last processed data file and its original image (if any):
ncview  /projects/arpae/terraref/sites/ua-mac/Level_1/hyperspectral_manualcheck/2016-12-11/2016-12-11__14-17-02-040/52079867-350a-48a6-9ef3-2f6049cc7b0f_.nc &
panoply /projects/arpae/terraref/sites/ua-mac/Level_1/hyperspectral_manualcheck/2016-12-11/2016-12-11__14-17-02-040/52079867-350a-48a6-9ef3-2f6049cc7b0f_.nc &
open /tmp/tmpwhg88m/52079867-350a-48a6-9ef3-2f6049cc7b0f_image.jpg
Elapsed time 0m42s

I'll check to make sure my code here:
https://github.com/terraref/extractors-hyperspectral/tree/master/hyperspectral
...matches any updates you have done in computing-pipeline, but in the meantime does this error mean anything to you? The _raw file was only 400 MB, so I'd like to get your thoughts before trying on e.g. a 102 GB _raw file.
But that aside, I prioritized SWIR and VNIR data in the Clowder rebuild and they've all been recreated as datasets, so after this is ironed out I'll run the extractor on all of them at long last.

Hi Max,
Thanks for testing this. The failure appears to be a false positive check emitted by the QAQC program. "The height of the image should always be 169 pxl" is something that @FlyingWithJerome apparently checks for in QAQC, but there is really no reason why an image should have 169 scanlines. We expect y = height = scanlines to vary. Jerome, please disable or alter this particular test if that is indeed the problem. FYI, I expect more false positives and small glitches now that we are running the workflow on more images. But they should be easy to solve.
@max-zilla the workflow is pretty flexible about directory structures. In particular, one can instruct it to write temporary files to any directory with -T $tmp_dir. This overrides both the default on roger, which is /gpfs_scratch/arpae/imaging_spectrometer, and the fallback default of $TMPDIR.

@czender @FlyingWithJerome thanks Charlie. The directory problem in this case was that if one runs this extractor out on a random server that can't access the files locally, it has to download them from Clowder first - Clowder creates a certain directory structure to avoid filename collisions that wasn't compatible with the workflow script (it wouldn't find every file in one folder, for example) so I made sure to handle that in my code.
With that in mind, I think the next step will be for me to queue and run a handful of variously sized input files and then share the outputs with you so we can make sure they look good on ~10-20 datasets before we run the rest.

"The _raw file was only 400 MB, so I'd like to get your thoughts before trying on e.g. a 102 GB _raw file." @max-zilla @solmazhajmohammadi @dlebauer @hmb1 We've discussed before what the maximum expected raw filesize would be and my recollection is that Lemnatec said 64 GB is the size of a raw file produced by a full field-width scan. The workflow works (I have tested it many times) on 64 GB raw files. The workflow will fail miserably with any significantly larger. That is what to expect. I don't know where the disconnect occurred, but somehow the maximum filesize is now 50% larger than the workflow was designed to handle. Until and unless the workflow is re-engineered, run it only on files smaller than ~65 GB :) Since there currently appears to be  no  maximum filesize, @hmb1 please alter the workflow to exit gracefully when the raw input file size, i.e., size of $in_fl in hyperspectral_workflow.sh, exceeds 65 GB? A few lines of Bash should do it.

That is fine with me - the large file was a SWIR file from early May so it could very well be deprecated at this point:
https://terraref.ncsa.illinois.edu/clowder/datasets/58713c954f0cc129fb5b88c7

Getting the extractor to work on data from May is a very low priority ... so sticking with the 64GB max size is reasonable.

@czender For a single row scan, the file size should not exceed the 64 GB(when we are running with minimum speed of 0.02m/s).
It might be just a test on the system to see if it is able to handle the larger file size, or system was running in free run mode(for the whole field).
Anyhow, If there is a file size larger than 64GB, please let me know the date and time. I can check it.

@czender @FlyingWithJerome OK, an update.
summary: extractor successfully triggered on VNIR file. still getting error on SWIR file.
VNIR
https://terraref.ncsa.illinois.edu/clowder/datasets/587024414f0c0dbad1a78b83
.nc file here, the output is on Roger at
/projects/arpae/terraref/sites/ua-mac/Level_1/hyperspectral_manualcheck/2016-12-11/2016-12-11__14-17-02-040/

output is 1.3 GB from a 400mb input.
SWIR
https://terraref.ncsa.illinois.edu/clowder/datasets/58713f394f0cc129fb5c2436
Tried triggering on this dataset from 12/08, but get this error:
Terraref hyperspectral data workflow invoked with:
hyperspectral_workflow.sh -d 1 -i /projects/arpae/terraref/sites/ua-mac/raw_data/SWIR/2016-12-08/2016-12-08__15-27-55-795/5254f64d-279c-47e5-843f-232f6777d0ef_2016_12_08_15_31_11raw -o /projects/arpae/terraref/sites/ua-mac/Level_1/hyperspectral_manualcheck/2016-12-08/2016-12-08__15-27-55-795/5254f64d-279c-47e5-843f-232f6777d0ef_2016_12_08_15_31_11.nc
Hyperspectral workflow scripts in directory /gpfs/largeblockFS/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral
NCO version "4.6.2-beta03" from directory /gpfs/smallblockFS/sw/nco-4.6.2-beta03/bin
Intermediate/temporary files written to directory /gpfs_scratch/arpae/imaging_spectrometer
Final output stored in directory /projects/arpae/terraref/sites/ua-mac/Level_1/hyperspectral_manualcheck/2016-12-08/2016-12-08__15-27-55-795
Input #00: /projects/arpae/terraref/sites/ua-mac/raw_data/SWIR/2016-12-08/2016-12-08__15-27-55-795/5254f64d-279c-47e5-843f-232f6777d0ef_2016_12_08_15_31_11raw
trn(in)  : /projects/arpae/terraref/sites/ua-mac/raw_data/SWIR/2016-12-08/2016-12-08__15-27-55-795/5254f64d-279c-47e5-843f-232f6777d0ef_2016_12_08_15_31_11raw
trn(out) : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_trn.nc.pid9776.fl00.tmp
ERROR: Unable to identify camera type (SWIR or VNIR?)
HINT: hyperspectral_workflow.sh requires header file  to report either 272 (SWIR) or 955 (VNIR) or wavelengths. Actual number reported = wvl_nbr = .

/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/hyperspectral_workflow.sh: line 520: [: : integer expression expected
/projects/arpae/terraref/shared/extractors/extractors-hyperspectral/hyperspectral/hyperspectral_workflow.sh: line 522: [: : integer expression expected

It says "unable to identify camera type". I'll let you take a look to see if you think your script needs to be changed, but I will also say that I can easily tell which it is by the dataset name/path so if it's easy to make that into one of the args I can provide when I call the script, that's fine too.
If the VNIR seems fine to you, I'll get started processing the rest!

@max-zilla the VNIR results look good.
I will fix the camera-type check so it assigns wvl_nbr == 273 to SWIR camera when I figure-out how to create a branch and a pull request. For now, what puzzles me is that your text output pasted above does not contain any printed variables. In other words, it does not contain the printed value of $wvl_nbr. So two things appear to need fixing. But SWIR is not operational, right? So please proceed with VNIR.

Going to close this issue now that the components are working at a basic level, with further adjustments going into https://github.com/terraref/computing-pipeline/issues/230


Iteration 2

Description
Next iteration on the API to expose more data, and refactor the model layer for adding additional data sources in near future.
Tasks
Model/Serialiser Refactor

[x] Remove direct dependency on django-cities models

[x] Data migration writes from Django Cities to our own tables/models
[x] CosmopolitanContinent
[x] CosmopolitanCountry
[x] CosmopolitanCurrency
[x] CosmopolitanCity
[x] CosmopolitanRegion
[ ] CosmopolitanPostCode

[x] Serialisers use these new models
[x] The primary key for each model should be a field explicitly called id, so we expose a standard interface even though the "primary key" has various different names like "code" or "name" in our data sources
[x] Update test suite to test our own models/serialisers

New endpoints
Cities

[ ] City List /api/v1/countries/ (/api/v1/cities/?countries={id}, /api/v1/cities/?countries={id},{id},{id})

Can filter by country ids, or list of country ids
Fields per object:

id - should be the ascii name?
name
name_std
url
kind
country (an object of code and url)
region (an object of code and url)

[ ] City Detail /api/v1/cities/{id}

Fields:

ALL the fields directly declared on the City model
country, being an object with the same serialiser as the country list object
region, being an object with the same serialiser as the region list object

Regions

[ ] Region List /api/v1/regions/ (/api/v1/regions/?countries={id}, /api/v1/regions/?countries={id},{id},{id})

Can filter by country ids, or list of country ids
Fields per object:

name
name_std
url
code
country (an object of code and url)

[ ] Region Detail /api/v1/regions/{id}

Fields:

ALL the fields directly declared on the City model
country, being an object with the same serialiser as the country list object
continent, being an object with the same serialiser as the region list object

Postcodes

[ ] Postcode List /api/v1/postcodes/ (/api/v1/postcodes/?countries={id}, /api/v1/postcodes/?countries={id},{id},{id})

Can filter by country ids, or list of country ids
Fields per object:

id
name_std
url
kind
country (an object of code and url)
region (an object of code and url)

[ ] Postcode Detail /api/v1/postcodes/{id}

Fields:

ALL the fields directly declared on the PostalCode model
country, being an object with the same serialiser as the country list object
region, being an object with the same serialiser as the region list object

@kiote do cities have AlternativeName references? If yes, we'll need to provide lookups based on that data, and also include it in serialisers. Please advise.

also as mentioned in the slack discussion, we need to

[ ] change migrations to custom management commands

@kiote yes, just data migrations, not migrations for Models/Schemas

do cities have AlternativeName references? If yes, we'll need to provide lookups based on that data, and also include it in serialisers.

Every model: City, Country, etc have AlternativeName as ManyToMany connection, so we can use that as well (but AltNames are not imported yet).
So we possibly able to add to every detail endpoint this:
alt_names = ['name',...,'name']

For Postal Code model:
django-cities does not have a link to region there, just a string representation of region name and they could even not exist: https://github.com/coderholic/django-cities/blob/master/cities/models.py#L132
For now in our model I left it as it is, what would you suggest?

I'd suggest it be a foreign key to a region, which is part of our model.

okay, but what to do with non-existing and empty values? skip them?

sure, it means the field is nullable.

seems like this one is done?


Better errors when skipping bad definitions

If I have:
module M = ...some valid definition...

module M = F(X)

let _ = M

Lets assume that [X] does not satisfy the signature that [F]
expects, i.e. the functor application is broken. When one asks
for the type of [M] in the last line, the type of the original
definition is reported.
This is very confusing. Ideally, merlin should report some error
saying that an intermediate definition of [M] was malformed,
instead of silently reporting the wrong type.
Value definitions seem to work a bit differently.  If I have the
followng snippet:
let f = 10

let f = () ()

let _ = f

and we ask for the type of [f] we see ['a]. In this case merlin
remembers the fact that it saw an [f] with a bad type.
But if we just had a syntax error:
let f = 10

let f = (

let _ = f

we now see [int].
Its a bit confusing that this happens for the same reasons. If
the two definitions of [f] are pretty far apart this gets really
hard to track down.
Like with modules, even though the second definition of [f] is
malformed, it would be nice to know that any later reference is
not to the original [f] but to some unknown thing.

Hi!
This thread is pretty old, and the example which motivated the issue
module M = ...
module M = F(X (* erroneous *))
let _ = M

is nowadays handled as expected (M on the last line has the return type of F).
Things are overall much better (more precise type recovery, much better syntax recovery), though they can still be somewhat improved.
But I think this issue can safely be closed.

Correct me if I am wrong, but didn't merlin start as a
modified ocaml top-level/REPL with the evaluator culled out?

Nope :)

Correct me if I am wrong, but didn't merlin start as a
modified ocaml top-level/REPL with the evaluator culled out?

Nope :)

Well, for a few days in late december 2012, it was :).


[WIP] Consolidate service interaction code between DPG, MPG and Legacy Data Plane

This issue contains description of the differences in client generation between autorest.csharp flavors and possible ways of consolidation. Description will be updated
Clients are different not only in terms of supported features, but also in how many types of clients is generated and how responsibilities are shared between them

DPG has only one client per endpoint. It contains all CreateXxxRequest methods, private response classifiers, sync and async version of the protocol methods and convenience methods that serialize request payload, call protocol methods and deserialize response (with the exception of Pageable, where convenience method calls create request method directly due to performance concerns).
Legacy Data Plane has internal RestClient per endpoint and may have public client that uses it. RestClient contains create request methods that serialize payload and sync/async convenience methods that handle response deserialization that may vary based in status codes. It also supports internal response header groups. Public client wraps RestClient method calls into client diagnostics scope and creates LRO instances. RestClient can have protocol methods similar to DPG, which would use their own version of create request methods.
MPG has internal RestClient per endpoint and multiple ArmResource- and ArmCollection-derived clients. RestClient is similar to Legacy Data Plane. Clients that use it may call one and the same method but with different set of parameters and/or has additional logic to create instance of ArmResource client that would contain deserialized resource data.

Summary of supported and non-supported features

Feature
DPG
Legacy Data Plane
MPG

Number of clients
Single public client
one internal RestClient and one public client
One internal rest client and one or more public ArmResource- and ArmCollection`-derived clients

Subclient / Resource client support
Yes
No
Yes

Parameter spread support
Yes
No
No

Parameter flattening support
No
Yes
No

Parameter grouping support
No
Yes
Yes

Request body serialization
Convenience methods
Create request methods
Create request methods

Multipart body support
No
Only in convenience methods
Yes

Multiple non-error response status code support
No
Yes
Yes

Response classifier support
Yes
Only in protocol methods
No

Header model
No
Yes
No

LRO types
Operation or Operation<T>
Generated LRO types, per client operation
Generated LRO types, one per client

LRO + Paging
No
Limited
Limited

Feature descriptions

Parameter spread: request payload is represented in convenience method signature by the set of parameters, induced from the properties of the payload object.
Parameter flattering: request payload is represented in convenience method signature by the set of parameters, induced from the properties of the flattened model
Parameter grouping: several elements of the request are represented in method signature as single parameter
Header model: internal type that caches values of the response headers collection
Request body serialization: logic that creates an instance of one of the types derived from RequestContent
Response classifier: type derived from Azure.Core.ResponseClassifier that provides information about response. Each protocol method has default response classifier, which can be overridden by caller via RequestContext

Consolidation challenges
Existing methods
Combined from all types of clients, there are 7 types of methods for non-LRO, non-paging case:

Legacy public client method

Validates parameters
Calls convenience rest client method with matching signature, wrapped in client diagnostics scope
Returns result as is

MPG public client method:

Validates parameters
Calls convenience rest client method with matching or more generic signature, wrapped in client diagnostics scope
Returns instance of ArmResource that contains new resource data

DPG convenience method

Validates parameters
Creates request content instance
Calls matching protocol method, wraps the call into client diagnostics scope if method name is different
Deserializes raw response into model
Returns typed response

MPG / Legacy convenience rest client method

Validates parameters
Creates message by calling rich CreateXxxRequest method
Calls _pipeline.Send/_pipeline.SendAsync to get untyped response
Deserializes response based on the status code
Returns typed response

DPG / Legacy protocol method

Validates parameters
Creates client diagnostic scope
Creates message by calling simple CreateXxxRequest method
Calls _pipeline.ProcessMessage/_pipeline.ProcessMessageAsync extension methods to get untyped response (these methods also call ResponseClassifier.IsErrorResponse)
Returns untyped response

Rich CreateXxxRequest method creates HttpMessage, creates RequestContent instance, handles parameter grouping and flattening
Simple CreateXxxRequest method only creates HttpMessage from parameters

Working in https://github.com/Azure/autorest.csharp/pull/3726


Dashboard not scaled on 4k resolution

Description
Dashboard is small on 4k resolution
After remove this line problem disappear
Environment
Gnome 44, Gentoo linux
Hardware
HW: https://linux-hardware.org/?probe=95231653d0
Installation
ALVR Version: https://github.com/alvr-org/ALVR/commit/92ec3d04a9d4ee4a553af1240a41bdb5ddccc8ad
Install Type:

[ ] Packaged (exe, deb, rpm, etc)
[ ] Portable (zip)
[*] Source

OS Name and Version: Gentoo Linux

This was a workaround for the steam deck the other solution would be to use Wayland

Which I think can be done with this WINIT_UNIX_BACKEND=wayland

Eh that's not necessary anymore it should now only do it on the Steamdeck

In fact this can be closed